diff --git a/COVERAGE_GAP.md b/COVERAGE_GAP.md index 5556cce8e..58a4e206b 100644 --- a/COVERAGE_GAP.md +++ b/COVERAGE_GAP.md @@ -79,3 +79,43 @@ Process: per-stage plan → implement → review, same as FIXED_ARRAY_REWORK.md. - Nightly cron on daspkg-index building every index package against daslang master — external ABI breakage surfaces as a nightly signal instead of inside an unrelated PR's extended_checks. + +## Stage 2 follow-ups (found while building utils/preflight, 2026-06-11) + +- **`.das_package` examples need a CLEAN daspkg install to be testable.** + `examples/games/sequence` resolves requires against a gitignored local + install (`modules/das-cards`) that only the smoke script refreshes — a + stale copy produced phantom compile errors (missing `card_mesh_ttf_path`) + against a green master. Options, discussable: (a) preflight scans for + `.das_package`, daspkg-installs fresh, then compile-checks — makes a + verify tool mutate + hit the network; (b) keep install-then-test inside + each example's smoke script (status quo, sequence only); (c) a separate + CI job that fresh-installs and tests every `.das_package`-bearing example + — pairs naturally with the Stage 4 nightly index cron. Leaning (c) with + preflight staying read-only. +- **JIT-to-exe BUNDLE exes fail to load locally while master CI is green.** + Two data points, same family: `exe_paths_module_resolve` ctest + (`uses_sqlite.exe`, 0xC0000135 DLL-not-found, bundle layout) and the + sequence smoke's bundled `sequence.exe --smoke` (0xC0000139 + entrypoint-not-found, fresh daspkg install + release, all artifacts + shipped). Triage so far: a trivial `daslang -exe` probe links AND runs + clean with the Dyn DLLs colocated; clearing `.jitted_scripts` changes + nothing; and a fully fresh one-shot rebuild (runtime DLLs + all shipped + modules + exe relinked together) still loads 0xC0000139 — so it is NOT + artifact staleness. Remaining suspect: the DLL-flavor daslang config + (local `.vscode` build) vs CI's static daslang, in the bundle exe's + load-time import chain. Chase both together. +- **Binary-staleness warning.** A `bin/.../daslang` older than the tree + produces convincing-but-wrong gate output (a stale binary's das2rst + regenerated handmade stubs under pre-rename `rtti` names). preflight + could compare the binary mtime against the newest `src/` commit and + WARN before running binary-derived gates. +- **Gate-duration baseline + regression warning.** Persist per-gate wall + times locally (gitignored) after each run; on the next run compare + against the recorded baseline and WARN when a gate is far off — ~10% + drift is noise, ~2x is indicative of a regression (compiler slowdown, + a lint rule gone pathological, doc-build growth). Care: per-changed- + file gates (lint, cpp-syntax) scale with the diff size, so normalize + per file or baseline only the fixed-cost gates (format sweep, dasgen, + docs, test suites). Pairs with the binary-staleness warning — both + are "preflight sanity-checks its own run" features. diff --git a/skills/filesystem.md b/skills/filesystem.md index 9a5d1d248..98946cd37 100644 --- a/skills/filesystem.md +++ b/skills/filesystem.md @@ -169,12 +169,13 @@ if (r is error) { ## Temp files & dirs ```das -let dir = create_temp_directory("build_") ?? "" // null/empty on failure -let f = create_temp_file("dump_", ".log") ?? "" -let td = temp_directory() ?? "" +var err : string +let dir = create_temp_directory("build_", err) // "" + err set on failure +let f = create_temp_file("dump_", ".log", err) +let td = temp_directory(err) ``` -All three return absolute paths; the `_result` variants surface a real error string if creation failed (disk full, permission denied, etc.). +All three return absolute paths (empty string on failure, with the `error` out-param set). There are NO zero/short-arg optional-returning forms — `temp_directory()` is a compile error. The `_result` variants (`temp_directory_result()`, `create_temp_file_result(prefix, ext)`, `create_temp_directory_result(prefix)`) wrap the same calls into `fs_result_string` for exhaustive matching. ## Disk space diff --git a/skills/make_pr.md b/skills/make_pr.md index fa4504749..2d3073400 100644 --- a/skills/make_pr.md +++ b/skills/make_pr.md @@ -2,6 +2,8 @@ Before creating a pull request, complete ALL of the following steps in order. Do not skip steps. If any step fails, fix the issue before proceeding. +**Shortcut:** `daslang utils/preflight/main.das -- --full` runs most of the mechanical gates below in one command (`skills/preflight.md` maps each gate to its CI lane). The steps here remain the authority on fix policy and on the judgment steps (dupe triage, workaround audit, doc stubs) the tool can't do. + ## 0. Sync with origin/master and rebase the branch **Always do this first.** If you skip it, a stale local `master` will cause your squashed commit to absorb other already-merged PRs as if they were branch-original work — the PR ends up touching files it has no business touching. diff --git a/skills/preflight.md b/skills/preflight.md index bcc201fc7..ead45f812 100644 --- a/skills/preflight.md +++ b/skills/preflight.md @@ -6,8 +6,14 @@ was an **oracle mismatch** — a gate CI enforces that no local step mirrored not a wrong change. This file maps every PR-triggered lane to its exact local mirror, or says honestly that there isn't one. -A `utils/preflight` tool automating the common tiers is planned -(`COVERAGE_GAP.md` Stage 2); until it lands, these are the manual commands. +**`utils/preflight` automates these gates.** `daslang utils/preflight/main.das` +runs the fast tier (format + lint + clang syntax pass on changed C++, seconds); +`-- --full` adds dasgen freshness, the CI-only-das compile sweep, the six doc +gates, ctest, the interp/JIT/AOT suites, and the sequence smoke. `--list-gates` +shows the menu; `--only ` / `--skip ` select subsets. Gates whose +host tool or module is missing report `SKIP` with an install/rebuild hint. The +tables below remain the reference for what each gate mirrors and for running +any step by hand. **Conventions.** `` = your compiler binary: `bin/Release/daslang.exe` (Windows MSVC multi-config), `bin/daslang` (Ninja single-config — what CI's diff --git a/utils/preflight/README.md b/utils/preflight/README.md new file mode 100644 index 000000000..d10969ea2 --- /dev/null +++ b/utils/preflight/README.md @@ -0,0 +1,32 @@ +# preflight + +Run CI's gates locally before pushing. The CI-lane ↔ gate mapping and the +manual commands this tool automates live in +[skills/preflight.md](https://github.com/GaijinEntertainment/daScript/blob/master/skills/preflight.md). + +```bash +# fast tier (seconds): format --verify, lint changed .das, clang syntax pass on changed C++ +daslang utils/preflight/main.das + +# full tier: adds dasgen freshness, CI-only-das compile sweep, the six doc +# gates, ctest -L small, interpreter/JIT/AOT suites, sequence smoke +daslang utils/preflight/main.das -- --full + +# subset / introspection +daslang utils/preflight/main.das -- --list-gates +daslang utils/preflight/main.das -- --only docs,ci-das +daslang utils/preflight/main.das -- --skip tests-aot --full +``` + +Cross-platform (Windows / macOS / Linux+WSL): subprocesses go through +`popen_argv` (no shell) — except the sequence gate, which by design runs CI's +own smoke scripts under pwsh/bash. The C++ syntax pass uses `clang-cl /Zs` on +Windows (preferring the VS-bundled clang — the same binary CI's ClangCL +toolset uses) and `clang -fsyntax-only` elsewhere, and a gate whose host tool +or module is missing reports `SKIP` with an install/rebuild hint instead of +passing silently. Exit code is non-zero when any gate fails. + +`ci_only_das.txt` lists the in-repo das surface that no default local build +compiles (dasOpenGL today); see the header comment there before adding +entries — surfaces that pull external daspkg packages belong to the +`sequence` gate, not the compile sweep. diff --git a/utils/preflight/ci_only_das.txt b/utils/preflight/ci_only_das.txt new file mode 100644 index 000000000..790811526 --- /dev/null +++ b/utils/preflight/ci_only_das.txt @@ -0,0 +1,12 @@ +# CI-only das surface: in-repo files no default local build compiles, swept by +# the preflight `ci-das` gate with `daslang -compile-only`. +# Format: | | +# A compile failure whose missing prerequisite is NOT in the allow-skip list is a FAIL. +# +# Only list files whose requires resolve entirely in-repo. Surfaces that pull +# external daspkg packages (examples/games/sequence -> das-cards) belong to the +# `sequence` gate instead — its ci_smoke_test script installs the deps fresh, +# while a bare -compile-only would read whatever stale gitignored install is +# lying around and report phantom errors. +modules/dasOpenGL/opengl/*.das | glfw, opengl | rebuild with -DDAS_GLFW_DISABLED=OFF +modules/dasOpenGL/glsl/*.das | glfw, opengl | rebuild with -DDAS_GLFW_DISABLED=OFF diff --git a/utils/preflight/main.das b/utils/preflight/main.das new file mode 100644 index 000000000..2f247ed19 --- /dev/null +++ b/utils/preflight/main.das @@ -0,0 +1,894 @@ +options gen2 +options indenting = 4 +options persistent_heap + +// preflight — run CI's gates locally before pushing. Gate ↔ CI-lane mapping +// and the manual commands this tool automates: skills/preflight.md. +// +// Fast tier (default): format --verify, lint on changed .das, clang +// syntax-only pass on changed C++. --full adds dasgen freshness, the +// CI-only-das compile sweep, the six doc gates, tests-cpp (ctest -L small), +// interpreter/JIT/AOT suites, and the sequence smoke. +// +// Cross-platform: Windows / macOS / Linux+WSL. Subprocesses go through +// popen_argv (no shell, no quoting trap), with one exception: the sequence +// gate runs CI's own smoke scripts under pwsh/bash by design. A gate whose +// host tool or module is missing reports SKIP with an install/rebuild hint, +// never a silent pass. + +require daslib/clargs +require daslib/fio +require daslib/strings_boost +require strings +require ../common/parallel_workers.das + + +[CommandLineArgs] +struct Config { + @clarg_doc = "Run the full tier (adds dasgen, ci-das, docs, tests-cpp, interp/JIT/AOT suites, sequence smoke)" + full : bool + + @clarg_doc = "Diff base for changed-file detection (default: origin/master)" + base : string = "origin/master" + + @clarg_doc = "Comma-separated gate subset to run (see --list-gates); overrides tier selection" + only : string + + @clarg_doc = "Comma-separated gates to skip" + skip : string + + @clarg_name = "daslang" + @clarg_doc = "daslang binary for child invocations (default: DASLANG env, then bin/, build/ probe)" + daslang_bin : string + + @clarg_doc = "CMake build directory (default: build)" + build_dir : string = "build" + + @clarg_doc = "clang / clang-cl binary for the C++ syntax pass (default: probe PATH)" + clang : string + + @clarg_short = "j" + @clarg_doc = "Parallel width for the C++ syntax pass; 0 = hardware threads" + jobs : int + + @clarg_doc = "Stop at the first failing gate" + fail_fast : bool + + @clarg_doc = "List gates with tier and description, then exit" + list_gates : bool + + @clarg_short = "v" + @clarg_doc = "Print child output for passing gates too" + verbose : bool + + @clarg_short = "?" + @clarg_name = "show-help" + @clarg_doc = "Show this help and exit" + help : bool +} + +enum GateStatus { + Pass + Fail + Skip +} + +struct GateResult { + name : string + status : GateStatus + seconds : double + detail : string // one-line summary: skip reason / failure hint + output : string // child output, printed on FAIL or --verbose +} + +struct PreflightCtx { + daslang : string + build_dir : string // "" when no configured build tree found + multi_config : bool + config : string // Release / Debug / RelWithDebInfo + clang : string // "" when not found + clang_is_cl : bool + base : string + changed_das : array + changed_cpp : array + changed_hdr : int + jobs : int + verbose : bool +} + +// ===== subprocess + small helpers ===== + +def run_argv(args : array; timeout : float) : tuple { + var output : string + let rc = unsafe(popen_argv(args, timeout) $(f) { + if (f != null) { + output := unsafe(fread_to_eof(f)) + } + }) + return (rc = rc, out = output) +} + +def run_argv(args : array) : tuple { + return run_argv(args, 0.0) +} + +def tool_available(exe, probe_flag : string) : bool { + let r = run_argv([exe, probe_flag], 30.0) + return r.rc == 0 +} + +def tool_available(exe : string) : bool { + return tool_available(exe, "--version") +} + +def seconds_since(t0 : int64) : double { + return double(get_time_usec(t0)) / 1000000.0lf +} + +def split_csv(s : string) : array { + var out : array + for (part in split(s, ",")) { + let p = strip(part) + if (!empty(p)) { + out |> push(p) + } + } + return <- out +} + +def temp_dir_or_cwd() : string { + var err : string + let td = temp_directory(err) + return empty(td) ? "." : td +} + +// unique per run — the system temp dir is shared across checkouts, so a +// fixed name lets concurrent preflights clobber each other's files +def unique_temp_path(prefix, suffix : string) : string { + return path_join(temp_dir_or_cwd(), "{prefix}_{ref_time_ticks()}{suffix}") +} + +def non_empty_lines(s : string) : array { + var out : array + for (raw in split(s, "\n")) { + let line = strip(replace(raw, "\r", "")) + if (!empty(line)) { + out |> push(line) + } + } + return <- out +} + +// ===== environment discovery ===== + +def find_daslang(cli_path : string) : string { + if (!empty(cli_path)) { + return fexist(cli_path) ? cli_path : "" + } + if (has_env_variable("DASLANG")) { + let env = get_env_variable("DASLANG") + if (fexist(env)) return env + } + let candidates = [ + "bin/daslang", "bin/daslang.exe", + "bin/Release/daslang.exe", "bin/RelWithDebInfo/daslang.exe", "bin/Debug/daslang.exe", + "build/daslang", "build/bin/daslang" + ] + for (c in candidates) { + if (fexist(c)) return c + } + return "" +} + +def detect_build_config(daslang : string) : string { + let generic = to_generic_path(daslang) + if (find(generic, "/Debug/") >= 0) return "Debug" + if (find(generic, "/RelWithDebInfo/") >= 0) return "RelWithDebInfo" + return "Release" +} + +def is_multi_config(build_dir : string) : bool { + let cache = fread(path_join(build_dir, "CMakeCache.txt")) + for (line in split(cache, "\n")) { + if (starts_with(line, "CMAKE_GENERATOR:INTERNAL=")) { + return find(line, "Visual Studio") >= 0 || find(line, "Xcode") >= 0 || find(line, "Multi-Config") >= 0 + } + } + return false +} + +def find_vs_clang_cl() : string { + // CI's clang-cl lane builds with the VS ClangCL toolset, i.e. the + // VS-bundled clang — the most CI-faithful binary on a Windows box. + // (A newer standalone LLVM emits diagnostics CI never sees: clang 22 + // rejects vecmath volatile-union passing that VS clang 19 accepts.) + var pf86 = "C:/Program Files (x86)" + if (has_env_variable("ProgramFiles(x86)")) { + pf86 = get_env_variable("ProgramFiles(x86)") + } + let vswhere = path_join(pf86, "Microsoft Visual Studio/Installer/vswhere.exe") + if (fexist(vswhere)) { + let r = run_argv([vswhere, "-latest", "-products", "*", "-find", "VC\\Tools\\Llvm\\x64\\bin\\clang-cl.exe"], 60.0) + if (r.rc == 0) { + let lines <- non_empty_lines(r.out) + if (!empty(lines) && fexist(lines[0])) return lines[0] + } + } + return "" +} + +def find_clang(cli_path, build_dir : string) : tuple { + if (!empty(cli_path)) { + let is_cl = find(base_name(cli_path), "clang-cl") >= 0 + return (exe = tool_available(cli_path) ? cli_path : "", is_cl = is_cl) + } + if (get_platform_name() == "windows") { + let vs_clang = find_vs_clang_cl() + if (!empty(vs_clang) && tool_available(vs_clang)) return (exe = vs_clang, is_cl = true) + if (tool_available("clang-cl")) return (exe = "clang-cl", is_cl = true) + // the prebuilt LLVM package dasLLVM fetches ships clang-cl next to + // LLVM.dll — present on any Windows box that built with dasLLVM, + // and /Zs needs no linker so the package's missing lld doesn't matter + if (!empty(build_dir)) { + let bundled = path_join(build_dir, "_deps/das_llvm_shared_lib-src/clang-cl.exe") + if (fexist(bundled) && tool_available(bundled)) return (exe = bundled, is_cl = true) + } + if (tool_available("clang")) return (exe = "clang", is_cl = false) + } else { + if (tool_available("clang++")) return (exe = "clang++", is_cl = false) + if (tool_available("clang")) return (exe = "clang", is_cl = false) + } + return (exe = "", is_cl = false) +} + +def collect_changed_files(base : string; var das_files, cpp_files : array&; var hdr_count : int&) { + var seen : table + let probe = run_argv(["git", "rev-parse", "--verify", "--quiet", base]) + var ranges : array + if (probe.rc == 0) { + ranges |> push("{base}...HEAD") + } else { + to_log(LOG_WARNING, "preflight: diff base '{base}' not found; only uncommitted changes are considered\n") + } + ranges |> push("HEAD") // staged + unstaged vs HEAD + for (r in ranges) { + let d = run_argv(["git", "diff", "--name-only", "--diff-filter=ACMR", r]) + continue if (d.rc != 0) + for (f in non_empty_lines(d.out)) { + continue if (key_exists(seen, f)) + seen |> insert(f) + continue if (!fexist(f)) + let ext = extension(f) + if (ext == ".das") { + das_files |> push(f) + } elif (ext == ".cpp" || ext == ".cc") { + cpp_files |> push(f) + } elif (ext == ".h" || ext == ".hpp") { + hdr_count ++ + } + } + } +} + +// ===== gate runners ===== + +def gate_format(ctx : PreflightCtx) : GateResult { + let t0 = ref_time_ticks() + // mirror .githooks/pre-push: verify TRACKED files only, so gitignored + // scratch that never reaches CI can't fail the gate + let ls = run_argv(["git", "ls-files", "--cached", "--", "*.das"]) + if (ls.rc != 0) { + return GateResult(name = "format", status = GateStatus.Fail, seconds = seconds_since(t0), + detail = "git ls-files failed", output = ls.out) + } + let list_file = unique_temp_path("preflight_fmt_list", ".txt") + if (!fwrite(list_file, ls.out)) { + return GateResult(name = "format", status = GateStatus.Fail, seconds = seconds_since(t0), + detail = "cannot write {list_file}") + } + let r = run_argv([ctx.daslang, "utils/das-fmt/dasfmt.das", "--", "--files-from", list_file, "--verify"]) + remove(list_file) + return GateResult(name = "format", status = r.rc == 0 ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), + detail = r.rc == 0 ? "" : "unformatted files — run the MCP format_file tool (or dasfmt without --verify)", + output = r.out) +} + +def gate_lint(ctx : PreflightCtx) : GateResult { + let t0 = ref_time_ticks() + if (empty(ctx.changed_das)) { + return GateResult(name = "lint", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "no .das files changed vs {ctx.base}") + } + var args <- [ctx.daslang, "utils/lint/main.das", "--"] + args |> push_from(ctx.changed_das) + args |> push("--quiet") + let r = run_argv(args) + return GateResult(name = "lint", status = r.rc == 0 ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), + detail = r.rc == 0 ? "{length(ctx.changed_das)} file(s) clean" : "lint warnings — fix or nolint each (see skills/make_pr.md §1)", + output = r.out) +} + +def gate_cpp_syntax(ctx : PreflightCtx) : GateResult { + let t0 = ref_time_ticks() + if (empty(ctx.changed_cpp)) { + let note = ctx.changed_hdr > 0 ? " — {ctx.changed_hdr} changed header(s) NOT validated by this gate; compile their includers (cmake --build) or rely on CI's clang lanes" : "" + return GateResult(name = "cpp-syntax", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "no .cpp files changed vs {ctx.base}{note}") + } + if (empty(ctx.clang)) { + return GateResult(name = "cpp-syntax", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "clang not found — install LLVM (clang-cl/clang on PATH) or pass --clang ") + } + // in-scope TUs: core + tests-cpp. Module/util TUs need external or + // build-generated headers; CI's clang-cl lane covers those. + var in_scope : array + var skipped : array + for (f in ctx.changed_cpp) { + let g = to_generic_path(f) + if (starts_with(g, "src/") || starts_with(g, "include/") || starts_with(g, "tests-cpp/")) { + in_scope |> push(f) + } else { + skipped |> push(f) + } + } + if (empty(in_scope)) { + return GateResult(name = "cpp-syntax", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "changed C++ is outside src/include/tests-cpp ({join(skipped, ", ")}) — needs module/external deps; no local clang gate compiles it, CI's clang-cl lane does") + } + var flags : array + if (ctx.clang_is_cl) { + flags <- ["/Zs", "/EHsc", "/std:c++17", "/W0"] + } else { + flags <- ["-fsyntax-only", "-std=c++17", "-w"] + } + let includes = ["include", "3rdparty/fmt/include", "3rdparty/uriparser/include", "tests-cpp/3rdparty"] + let workers = compute_worker_count(length(in_scope), ctx.jobs) + let chunks <- chunk_files(in_scope, workers) + var argvs : array> + argvs |> reserve(length(chunks)) + for (chunk in chunks) { + var av : array + av |> reserve(1 + length(flags) + length(includes) + length(chunk)) + av |> push(ctx.clang) + av |> push_from(flags) + for (inc in includes) { + av |> push("-I{inc}") + } + av |> push_from(chunk) + argvs |> emplace(av) + } + var results <- run_chunk_workers(argvs) + var failed = false + for (r in results) { + if (r.exit_code != 0) { + failed = true + } + } + let output = build_string() $(w) { + for (r in results) { + if (!empty(r.stdout)) { + w |> write(r.stdout) + w |> write("\n") + } + } + } + var detail = "{length(in_scope)} file(s) via {ctx.clang}" + if (!empty(skipped)) { + detail = "{detail}; out of scope: {join(skipped, ", ")}" + } + return GateResult(name = "cpp-syntax", status = failed ? GateStatus.Fail : GateStatus.Pass, + seconds = seconds_since(t0), detail = detail, output = output) +} + +def gate_dasgen(ctx : PreflightCtx) : GateResult { + let t0 = ref_time_ticks() + let r = run_argv([ctx.daslang, "dasgen/gen_bind.das"]) + if (r.rc != 0) { + return GateResult(name = "dasgen", status = GateStatus.Fail, seconds = seconds_since(t0), + detail = "gen_bind.das failed", output = r.out) + } + let d = run_argv(["git", "diff", "--exit-code", "--stat", "--", "include/daScript/builtin/"]) + return GateResult(name = "dasgen", status = d.rc == 0 ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), + detail = d.rc == 0 ? "" : "generated .inc files out of date — commit the regenerated files", + output = d.out) +} + +// ===== docs gates (mirror doc.yml 1-6) ===== + +def skip_sphinx_gates(reason : string; var results : array&) { + let names = ["docs/sphinx-latex", "docs/sphinx-html"] + results |> reserve(length(results) + length(names)) + for (g in names) { + results |> emplace(GateResult(name = g, status = GateStatus.Skip, detail = reason)) + } +} + +def docs_skip_all(reason : string; var results : array&) { + let names = ["docs/das2rst", "docs/stubs", "docs/uncategorized", "docs/untracked", "docs/sphinx-latex", "docs/sphinx-html"] + results |> reserve(length(results) + length(names)) + for (g in names) { + results |> emplace(GateResult(name = g, status = GateStatus.Skip, detail = reason)) + } +} + +def gate_docs(ctx : PreflightCtx; var results : array&) { + var t0 = ref_time_ticks() + // gate 1: das2rst runs clean (positional handmade-doc validation) + let r1 = run_argv([ctx.daslang, "doc/reflections/das2rst.das"]) + if (r1.rc != 0 && find(r1.out, "missing prerequisite") >= 0) { + docs_skip_all("daslang lacks HV/PUGIXML — rebuild with -DDAS_HV_DISABLED=OFF -DDAS_PUGIXML_DISABLED=OFF", results) + return + } + results |> emplace(GateResult(name = "docs/das2rst", status = r1.rc == 0 ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), + detail = r1.rc == 0 ? "" : "handmade-doc validation panic — fix and re-run; CI stops at the FIRST panic, more may follow", + output = r1.out)) + + // gate 2: no `// stub` in handmade docs + t0 = ref_time_ticks() + var stubs : array + dir("doc/source/stdlib/handmade") $(name) { + return if (extension(name) != ".rst") + let p = path_join("doc/source/stdlib/handmade", name) + if (find(fread(p), "// stub") >= 0) { + stubs |> push(p) + } + } + results |> emplace(GateResult(name = "docs/stubs", status = empty(stubs) ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), + detail = empty(stubs) ? "" : "stub docs need real text (skills/make_pr.md §4c)", + output = join(stubs, "\n"))) + + // gate 3: no Uncategorized sections in generated docs + t0 = ref_time_ticks() + var uncategorized : array + dir("doc/source/stdlib/generated") $(name) { + return if (extension(name) != ".rst") + let p = path_join("doc/source/stdlib/generated", name) + for (line in split(fread(p), "\n")) { + if (strip(replace(line, "\r", "")) == "Uncategorized") { + uncategorized |> push(p) + break + } + } + } + results |> emplace(GateResult(name = "docs/uncategorized", status = empty(uncategorized) ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), + detail = empty(uncategorized) ? "" : "add the function(s) to a group_by_regex in doc/reflections/das2rst.das", + output = join(uncategorized, "\n"))) + + // gate 4: no untracked generated RST + t0 = ref_time_ticks() + let r4 = run_argv(["git", "ls-files", "--others", "--exclude-standard", "doc/source/stdlib/"]) + let untracked <- non_empty_lines(r4.out) + let untracked_ok = r4.rc == 0 && empty(untracked) + results |> emplace(GateResult(name = "docs/untracked", status = untracked_ok ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), + detail = untracked_ok ? "" : (r4.rc != 0 ? "git ls-files failed (rc={r4.rc})" : "das2rst generated new files — git add them"), + output = r4.rc != 0 ? r4.out : join(untracked, "\n"))) + + // gates 5+6: sphinx -W, latex then html. Delete the doctree cache first — + // CI builds from a fresh clone, a stale local cache hides warnings. + if (!tool_available("sphinx-build")) { + skip_sphinx_gates("sphinx-build not found — pip install -r doc/requirements.txt", results) + return + } + rmdir_rec("doc/sphinx-build") + if (stat("doc/sphinx-build").is_valid) { + // running sphinx against a cache we failed to delete would be an + // untrustworthy PASS — refuse instead + skip_sphinx_gates("cannot delete doc/sphinx-build (locked?) — remove it and re-run", results) + return + } + t0 = ref_time_ticks() + let r5 = run_argv(["sphinx-build", "-W", "--keep-going", "-b", "latex", "-d", "doc/sphinx-build", "doc/source", "build/latex"]) + results |> emplace(GateResult(name = "docs/sphinx-latex", status = r5.rc == 0 ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), detail = r5.rc == 0 ? "" : "latex builder warnings (warnings-as-errors)", output = r5.out)) + t0 = ref_time_ticks() + let r6 = run_argv(["sphinx-build", "-W", "--keep-going", "-b", "html", "-d", "doc/sphinx-build", "doc/source", "build/site"]) + results |> emplace(GateResult(name = "docs/sphinx-html", status = r6.rc == 0 ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), detail = r6.rc == 0 ? "" : "html builder warnings (warnings-as-errors)", output = r6.out)) +} + +// ===== CI-only das compile sweep ===== + +def gate_ci_das(ctx : PreflightCtx) : GateResult { + let t0 = ref_time_ticks() + let list_path = "utils/preflight/ci_only_das.txt" + let list_text = fread(list_path) + if (empty(list_text)) { + return GateResult(name = "ci-das", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "{list_path} not found or empty") + } + var checked = 0 + var skipped_mods : table + var fail_parts : array + for (raw_line in split(list_text, "\n")) { + let line = strip(replace(raw_line, "\r", "")) + continue if (empty(line) || starts_with(line, "#")) + let parts <- split(line, "|") + continue if (length(parts) < 3) + let pattern = strip(parts[0]) + let allow_skip <- split_csv(parts[1]) + let hint = strip(parts[2]) + var files : array + expand_glob(pattern, files) + if (empty(files)) { + fail_parts |> push("{pattern}: no files matched (stale ci_only_das.txt entry?)\n") + continue + } + for (f in files) { + let r = run_argv([ctx.daslang, "-compile-only", f], 300.0) + if (r.rc == 0) { + checked ++ + continue + } + // a missing NATIVE module from the entry's allow-skip list means the + // local binary was built without it — environment gap, not a failure + var module_gap = "" + for (m in allow_skip) { + if (find(r.out, "missing prerequisite '{m}'") >= 0 || find(r.out, "module {m} not found") >= 0) { + module_gap = m + break + } + } + if (!empty(module_gap)) { + if (!key_exists(skipped_mods, "{module_gap} ({hint})")) { + skipped_mods |> insert("{module_gap} ({hint})") + } + } else { + fail_parts |> push("=== {f} ===\n{r.out}\n") + } + } + } + var detail = "{checked} file(s) compile-checked" + if (!empty(skipped_mods)) { + let mods <- [for (k in keys(skipped_mods)); k] + let mods_text = join(mods, "; ") + detail = "{detail}; skipped entries needing: {mods_text}" + } + if (!empty(fail_parts)) { + return GateResult(name = "ci-das", status = GateStatus.Fail, seconds = seconds_since(t0), + detail = "compile errors in CI-only das surface", output = join(fail_parts, "")) + } + return GateResult(name = "ci-das", status = checked > 0 ? GateStatus.Pass : GateStatus.Skip, + seconds = seconds_since(t0), detail = detail) +} + +// ===== test suites ===== + +def run_test_gate(name : string; args : array; fail_hint : string) : GateResult { + let t0 = ref_time_ticks() + let r = run_argv(args, 3600.0) + return GateResult(name = name, status = r.rc == 0 ? GateStatus.Pass : GateStatus.Fail, + seconds = seconds_since(t0), detail = r.rc == 0 ? "" : fail_hint, output = r.out) +} + +// build-gate failure triage. LNK1104 on bin outputs = a live daslang process +// holds the file (make_pr §3). The unavoidable case: a DLL-flavor daslang +// import-links libDaScriptDyn*.dll, so the daslang HOSTING preflight pins +// them and no child build can ever relink — SKIP with a two-step hint +// (CI's static daslang doesn't have this; fresh targets skip the relink). +def classify_build_fail(gate, base : string; r : tuple; daslang : string; t0 : int64) : GateResult { + var detail = base + var status = GateStatus.Fail + if (find(r.out, "LNK1104") >= 0) { + if (fexist(path_join(dir_name(daslang), "libDaScriptDyn_runtime.dll"))) { + status = GateStatus.Skip + detail = "DLL-flavor daslang pins libDaScriptDyn*.dll, so preflight's own host blocks the relink — build the gate's targets from a plain shell once, then re-run (fresh targets skip the relink)" + } else { + detail = "{base} — LNK1104: a running daslang/daslang-live/MCP process is locking build outputs; stop it and re-run" + } + } + return GateResult(name = gate, status = status, seconds = seconds_since(t0), detail = detail, output = r.out) +} + +def gate_tests_interp(ctx : PreflightCtx) : GateResult { + return run_test_gate("tests-interp", + [ctx.daslang, "dastest/dastest.das", "--", "--failures-only", "--timeout", "900", "--test", "tests"], + "interpreter suite failures") +} + +def gate_tests_jit(ctx : PreflightCtx) : GateResult { + let t0 = ref_time_ticks() + // probe: a JIT no-op run tells us whether dasLLVM is in this binary + let probe_file = unique_temp_path("preflight_jit_probe", ".das") + if (!fwrite(probe_file, "options gen2\n[export]\ndef main() \{\n print(\"ok\\n\")\n\}\n")) { + return GateResult(name = "tests-jit", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "cannot write JIT probe file {probe_file}") + } + let probe = run_argv([ctx.daslang, "-jit", probe_file], 600.0) + remove(probe_file) + if (probe.rc != 0) { + return GateResult(name = "tests-jit", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "JIT probe failed (see output) — most common cause: daslang built without dasLLVM (-DDAS_LLVM_DISABLED=OFF)", output = probe.out) + } + return run_test_gate("tests-jit", + [ctx.daslang, "dastest/dastest.das", "-jit", "--", "--jit-opt-level=3", "--failures-only", "--timeout", "900", "--test", "tests"], + "JIT suite failures") +} + +def gate_tests_cpp(ctx : PreflightCtx) : GateResult { + let t0 = ref_time_ticks() + if (empty(ctx.build_dir)) { + return GateResult(name = "tests-cpp", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "no configured build dir — cmake -B build first") + } + var args <- ["ctest", "--test-dir", ctx.build_dir] + if (ctx.multi_config) { + args |> push("--build-config") + args |> push(ctx.config) + } + args |> push("-L") + args |> push("small") + args |> push("--no-tests=error") // unbuilt tests-cpp-small must FAIL, not 0-tests-pass + args |> push("--output-on-failure") + return run_test_gate("tests-cpp", args, "ctest -L small failures (build tests-cpp-small target if missing)") +} + +def gate_tests_aot(ctx : PreflightCtx) : GateResult { + let t0 = ref_time_ticks() + if (empty(ctx.build_dir)) { + return GateResult(name = "tests-aot", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "no configured build dir — cmake -B build first") + } + let b = run_argv(["cmake", "--build", ctx.build_dir, "--config", ctx.config, "--target", "test_aot", "--parallel"]) + if (b.rc != 0) { + // root CMakeLists gates tests/aot out on DAS_AOT_EXAMPLES_DISABLED and + // 32-bit Windows — an absent target is a config choice, not a failure. + // Unknown-target spellings: MSBuild / Ninja / Make. + for (pat in ["MSB1009", "unknown target", "No rule to make target"]) { + if (find(b.out, pat) >= 0) { + return GateResult(name = "tests-aot", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "test_aot target not in this build config (DAS_AOT_EXAMPLES_DISABLED, or 32-bit Windows) — the AOT suite runs on 64-bit CI lanes", output = b.out) + } + } + return classify_build_fail("tests-aot", + "test_aot build failed (new test dirs need tests/aot/CMakeLists.txt registration)", b, ctx.daslang, t0) + } + var test_aot = path_join(dir_name(ctx.daslang), "test_aot") + if (!fexist(test_aot) && fexist("{test_aot}.exe")) { + test_aot = "{test_aot}.exe" + } + if (!fexist(test_aot)) { + return GateResult(name = "tests-aot", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "test_aot binary not found next to daslang ({test_aot})") + } + return run_test_gate("tests-aot", + [test_aot, "-use-aot", "dastest/dastest.das", "--", "--use-aot", "--failures-only", "--timeout", "900", "--test", "tests"], + "AOT suite failures (error[50101] → skills/aot_hash_desync_debugging.md)") +} + +def gate_sequence(ctx : PreflightCtx) : GateResult { + let t0 = ref_time_ticks() + if (empty(ctx.build_dir)) { + return GateResult(name = "sequence", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "no configured build dir — cmake -B build first") + } + let b = run_argv(["cmake", "--build", ctx.build_dir, "--config", ctx.config, "--target", + "dasModuleGlfw", "dasModuleLiveHost", "dasModuleHV", "dasModuleAudio", + "dasModulePUGIXML", "dasModuleStbImage", "--parallel"]) + if (b.rc != 0) { + let lower = to_lower(b.out) + if (find(lower, "unknown target") >= 0 || find(lower, "no rule to make target") >= 0 || find(lower, "does not exist") >= 0) { + return GateResult(name = "sequence", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "module targets missing — configure with release modules ON (ci/release_modules.txt)", output = b.out) + } + return classify_build_fail("sequence", "module build failed", b, ctx.daslang, t0) + } + let cwd = getcwd() + // both smoke scripts honor $BIN (multi-config layouts put daslang in + // bin/); daslang has no setenv, so inject via the shell + let bin_dir = dir_name(ctx.daslang) + var args : array + if (get_platform_name() == "windows") { + let shell = tool_available("pwsh") ? "pwsh" : "powershell" + // single-quoted PS literals: escape embedded quotes by doubling + let bin_dir_ps = replace(bin_dir, "'", "''") + let cwd_ps = replace(cwd, "'", "''") + let cmd = "$env:BIN = '{bin_dir_ps}'; & 'examples/games/sequence/ci_smoke_test.ps1' '{cwd_ps}'; exit $LASTEXITCODE" + args <- [shell, "-NoProfile", "-Command", cmd] + } else { + args <- ["env", "BIN={bin_dir}", "bash", "examples/games/sequence/ci_smoke_test.sh", cwd] + if (get_platform_name() == "linux" && !has_env_variable("DISPLAY")) { + // xvfb-run is a wrapper script: -h is its documented flag; --version + // support varies by distro version + if (tool_available("xvfb-run", "-h")) { + var wrapped <- ["xvfb-run", "-a"] + wrapped |> push_from(args) + args <- wrapped + } else { + return GateResult(name = "sequence", status = GateStatus.Skip, seconds = seconds_since(t0), + detail = "headless linux without xvfb-run — apt-get install xvfb") + } + } + } + return run_test_gate("sequence", args, "sequence smoke failed (examples/games/sequence/ci_smoke_test.*)") +} + +// ===== orchestration ===== + +struct GateInfo { + name : string + full_only : bool + doc : string +} + +def gate_table() : array { + return <- [ + GateInfo(name = "format", full_only = false, doc = "formatter --verify on tracked .das (mirrors pre-push + CI)"), + GateInfo(name = "lint", full_only = false, doc = "lint changed .das, zero warnings"), + GateInfo(name = "cpp-syntax", full_only = false, doc = "clang syntax-only pass on changed C++ (MSVC-vs-clang 80/20)"), + GateInfo(name = "dasgen", full_only = true, doc = "gen_bind.das freshness vs include/daScript/builtin/"), + GateInfo(name = "ci-das", full_only = true, doc = "compile-only sweep of CI-only das surface (ci_only_das.txt)"), + GateInfo(name = "docs", full_only = true, doc = "the six doc.yml gates (das2rst, stubs, uncategorized, untracked, sphinx ×2)"), + GateInfo(name = "tests-cpp", full_only = true, doc = "ctest -L small"), + GateInfo(name = "tests-interp", full_only = true, doc = "full interpreter suite"), + GateInfo(name = "tests-jit", full_only = true, doc = "full JIT suite (skips when dasLLVM absent)"), + GateInfo(name = "tests-aot", full_only = true, doc = "build test_aot + full AOT suite"), + GateInfo(name = "sequence", full_only = true, doc = "sequence game release smoke (only pre-merge compile of GLFW-gated das)") + ] +} + +def want_gate(cfg : Config; info : GateInfo) : bool { + let only_list <- split_csv(cfg.only) + let skip_list <- split_csv(cfg.skip) + for (s in skip_list) { + if (s == info.name) return false + } + if (!empty(only_list)) { + for (o in only_list) { + if (o == info.name) return true + } + return false + } + return cfg.full || !info.full_only +} + +def status_tag(s : GateStatus) : string { + if (s == GateStatus.Pass) return "PASS" + if (s == GateStatus.Fail) return "FAIL" + return "SKIP" +} + +def report_gate(r : GateResult; verbose : bool) { + let secs = fmt(":.1f", float(r.seconds)) + var timing = "" + if (r.seconds > 0.05lf) { + timing = " ({secs}s)" + } + var detail = "" + if (!empty(r.detail)) { + detail = " — {r.detail}" + } + let level = r.status == GateStatus.Fail ? LOG_ERROR : LOG_INFO + let tag = status_tag(r.status) + to_log(level, "[{tag}] {r.name}{timing}{detail}\n") + if ((r.status == GateStatus.Fail || verbose) && !empty(r.output)) { + to_log(level, "{r.output}\n") + } +} + +[export] +def main() : int { + var args_r <- parse_args(type) + if (args_r |> is_err) { + to_log(LOG_ERROR, "error: {args_r |> unwrap_err}\n\n") + print_help(get_command_info(type), "preflight") + return 1 + } + let cfg <- args_r |> move_unwrap + if (cfg.help) { + print_help(get_command_info(type), "preflight") + return 0 + } + if (cfg.list_gates) { + for (info in gate_table()) { + to_log(LOG_INFO, "{info.name}\t{info.full_only ? "full" : "fast"}\t{info.doc}\n") + } + return 0 + } + if (!fexist("dastest/dastest.das") || !fexist("CMakeLists.txt")) { + to_log(LOG_ERROR, "preflight must run from the daslang repo root (dastest/dastest.das not found in cwd)\n") + return 1 + } + + var ctx = PreflightCtx(base = cfg.base, jobs = cfg.jobs, verbose = cfg.verbose) + ctx.daslang = find_daslang(cfg.daslang_bin) + if (empty(ctx.daslang)) { + to_log(LOG_ERROR, "daslang binary not found — build it (cmake --build build --target daslang) or pass --daslang / set DASLANG\n") + return 1 + } + ctx.config = detect_build_config(ctx.daslang) + if (fexist(path_join(cfg.build_dir, "CMakeCache.txt"))) { + ctx.build_dir = cfg.build_dir + ctx.multi_config = is_multi_config(cfg.build_dir) + } + let clang_info = find_clang(cfg.clang, ctx.build_dir) + ctx.clang = clang_info.exe + ctx.clang_is_cl = clang_info.is_cl + collect_changed_files(cfg.base, ctx.changed_das, ctx.changed_cpp, ctx.changed_hdr) + + let tier = cfg.full ? "full" : "fast" + let n_das = length(ctx.changed_das) + let n_cpp = length(ctx.changed_cpp) + to_log(LOG_INFO, "preflight: {tier} tier; daslang={ctx.daslang}; base={cfg.base}; {n_das} das / {n_cpp} cpp changed\n") + + var results : array + let t_all = ref_time_ticks() + for (info in gate_table()) { + continue if (!want_gate(cfg, info)) + to_log(LOG_INFO, "[RUN ] {info.name}\n") + if (info.name == "docs") { + var doc_results : array + gate_docs(ctx, doc_results) + for (r in doc_results) { + report_gate(r, cfg.verbose) + } + results |> push_from(doc_results) + } else { + var r = GateResult(name = info.name) + if (info.name == "format") { + r <- gate_format(ctx) + } elif (info.name == "lint") { + r <- gate_lint(ctx) + } elif (info.name == "cpp-syntax") { + r <- gate_cpp_syntax(ctx) + } elif (info.name == "dasgen") { + r <- gate_dasgen(ctx) + } elif (info.name == "ci-das") { + r <- gate_ci_das(ctx) + } elif (info.name == "tests-cpp") { + r <- gate_tests_cpp(ctx) + } elif (info.name == "tests-interp") { + r <- gate_tests_interp(ctx) + } elif (info.name == "tests-jit") { + r <- gate_tests_jit(ctx) + } elif (info.name == "tests-aot") { + r <- gate_tests_aot(ctx) + } elif (info.name == "sequence") { + r <- gate_sequence(ctx) + } + report_gate(r, cfg.verbose) + results |> emplace(r) + } + var any_fail = false + for (r in results) { + if (r.status == GateStatus.Fail) { + any_fail = true + break + } + } + break if (cfg.fail_fast && any_fail) + } + + var n_pass = 0 + var n_fail = 0 + var n_skip = 0 + for (r in results) { + if (r.status == GateStatus.Pass) { + n_pass ++ + } elif (r.status == GateStatus.Fail) { + n_fail ++ + } else { + n_skip ++ + } + } + let total_s = fmt(":.1f", float(seconds_since(t_all))) + to_log(n_fail > 0 ? LOG_ERROR : LOG_INFO, + "\npreflight: {n_pass} passed, {n_fail} failed, {n_skip} skipped in {total_s}s\n") + for (r in results) { + if (r.status != GateStatus.Pass) { + let tag = status_tag(r.status) + var detail = "" + if (!empty(r.detail)) { + detail = " — {r.detail}" + } + to_log(r.status == GateStatus.Fail ? LOG_ERROR : LOG_INFO, " [{tag}] {r.name}{detail}\n") + } + } + return n_fail > 0 ? 1 : 0 +}