feat(buck2): distributed CI building + testing the whole tree on free runners#1862
Draft
gilescope wants to merge 32 commits into
Draft
feat(buck2): distributed CI building + testing the whole tree on free runners#1862gilescope wants to merge 32 commits into
gilescope wants to merge 32 commits into
Conversation
… runners Adds a buck2 build of midnight-node (parallel to cargo/bazel) that culminates in a distributed CI workflow: no paid 16-core box — a driver job owns buck2 and N ubuntu-latest worker jobs lend cores over a rebuck2 iroh P2P mesh. Build system - reindeer-generated third-party graph + first-party BUCK (scripts/buck2/*). - Real on-chain WASM runtime blob built end-to-end: wasm32 no_std crate universe (wasm-deps/) + first-party wasm variants via select() on cpu:wasm32, cdylib -> wasm-opt -> zstd (tools/wasm-compress) -> include_bytes! inject. 84 substrate exports; node builds a dev chainspec from it. - buck2 test root//... --exclude ci-infra: Pass 1460 / Fail 0 locally. Distributed CI (.github/workflows/buck2-distributed.yml) - rebuck2 driver + 3 workers, re-exec platform (platforms/defs.bzl), keyless rendezvous via github.run_id. - Portable wasm toolchain: toolchains/BUCK reads [wasm] cc/cxx/ar from config (macOS Homebrew LLVM defaults; CI overrides with distro clang), wasm-lld.sh locates rust-lld via the sysroot. .github/actions/buck2-linux-toolchain installs rustc+wasm32v1-none, clang/lld, wasm-opt. - The 5 Docker/devnet/Cardano tests carry a ci-infra label and are excluded. See docs/buck2-notes.md for the full design. Assisted-by: Claude:claude-opus-4.8
gilescope
added
skip-changes-check-all
ai-assisted
First CI run surfaced two Linux issues:
- re-exec (local_enabled=False) rejects reindeer's local_only git_fetch actions
("Trying to execute a local_only=True action on remote executor"). Switch to
re-cache (hybrid): local_only runs local, the rest can go to the mesh.
- buck2 host auto-detection misfired on the x86_64 runner — third-party platform
selects resolved to arm (blake3 neon.c on x86, stacker windows.c on Linux,
if-addrs missing libc, windows-result raw-dylib on ELF). Pin the build with
--target-platforms root//platforms:linux-x86_64 so os:linux+cpu:x86_64 selects
the correct branches deterministically. Verified locally: blake3 → simd_x86.
Also: arch diagnostic step + soften the distribution check to a report.
Assisted-by: Claude:claude-opus-4.8
This comment has been minimized.
This comment has been minimized.
Run #2 got past the platform + git_fetch fixes and deep into the compile, then failed: `buck2 build root//...` also builds orphan platform-specific object targets — ring's Windows `*-nasm.o` prebuilt_cxx_library targets, which reindeer emits from the fixup's `cfg(target_os="windows")` block WITHOUT `target_compatible_with`. So `root//...` doesn't skip them off-Windows and ld.lld fails "unknown file type" linking Windows NASM objects on Linux. These orphans are depended on by nothing on Linux (ring's lib correctly selects the elf asm). Build the actual products instead — node + toolkit binaries + the wasm blob — which pull the whole first-party/third-party closure through their real, platform-correct deps. `buck2 test` then builds+runs every test target and its deps (also not the orphans). This is the "builds and runs the tests" scope. Follow-up: teach reindeer to emit target_compatible_with for os-cfg-gated cxx targets so `root//...` is fully buildable. Assisted-by: Claude:claude-opus-4.8
…op 2.3GB snapshot Run #3 was a FALSE green. `buck2 … 2>&1 | tee X.log` returns tee's exit (0), so an 8-target build failure in the test step still passed the job. Real result was "Tests finished: Pass 1381. Fail 0. Build failure 8". - set -o pipefail on the build+test steps so buck2's non-zero exit fails the job. - The 8 failures were rustup, not code: rustc in the repo honours rust-toolchain.toml's pinned `components`, and parallel rustc actions race to rustup-install clippy on-the-fly -> "detected conflict: bin/cargo-clippy". Fix: pre-install those components in the composite (single-threaded) AND pin RUSTUP_TOOLCHAIN so rustc uses the installed 1.95 directly, never re-syncing. - snapshot-key-prefix: "" — the monolith store snapshot was 2.3 GB per run (blows the actions-cache budget). Cold builds until sharded-CAS persistence. Assisted-by: Claude:claude-opus-4.8
…ns it) Run #4 (now honestly failing): rustc still raced a rustup install — 'rust-std-aarch64-unknown-linux-gnu' conflict. rust-toolchain.toml pins targets = [wasm32v1-none, aarch64-unknown-linux-gnu]; the composite only pre-installed wasm32v1-none, so rustc auto-installed the aarch64 std under parallel actions and conflicted. RUSTUP_TOOLCHAIN didn't help — buck2 sanitizes the action env, so rustc still reads rust-toolchain.toml. Pre-install every pinned target so there's nothing left to sync at build time. The secp256k1-sys wasm build-script cc failure had empty clang stderr and is likely collateral from the racing rustup installs (rollback corrupts the shared toolchain mid-build); re-evaluating with a clean toolchain. Assisted-by: Claude:claude-opus-4.8
Run #5 (clean toolchain, real errors now): - openssl-0.10 "not found in `ffi`": openssl-sys's build script probes the system openssl to set version cfgs; without libssl-dev + pkg-config it sets the wrong ones and openssl's ffi bindings don't resolve. Install them. - secp256k1-sys 0.10.1 wasm.c fails on ubuntu clang-18: its vendored wasm-sysroot predates clang making implicit function declarations a hard error. Local fixup adds CFLAGS_wasm32v1_none=-Wno-error=implicit-function- declaration to the build-script env (symbols resolve at link). AGENTS.md documents this exact case. Verified: blob still builds on macOS. Assisted-by: Claude:claude-opus-4.8
Run #6: the -Wno-error CFLAG applied but secp256k1's wasm.c still failed (buck discards the build-script's clang stderr, so it looked mysterious). The real cause: the cc-rs command carries wasm codegen flags (-mcpu=mvp -mmutable-globals) but NO --target — cc-rs doesn't map the `wasm32v1-none` rust target name to a clang triple, so clang treats -mcpu=mvp as an x86 CPU and dies. Add --target=wasm32-unknown-unknown to the build-script CFLAGS (what secp256k1-sys 0.9 does itself, hence it built). Verified: blob still builds on macOS. Assisted-by: Claude:claude-opus-4.8
Adopt the fixes the sibling bazel build already uses for these -sys crates: - openssl (native-tls, Linux-only — macOS uses Security.framework): local openssl-sys fixup sets OPENSSL_LIB_DIR/OPENSSL_INCLUDE_DIR to the system paths so the build script probes the right version and emits consistent ossl3xx cfgs (fixes "not found in `ffi`"). Mirrors MODULE.bazel. - secp256k1-sys wasm.c: toolchains/wasm-cc.sh wrapper forces --target=wasm32-unknown-unknown (cc-rs doesn't map wasm32v1-none), resolving a wasm-capable clang; wasm-ar.sh for llvm-ar. Mirrors bazel wasm32_cc.sh. CI points [wasm] cc/cxx/ar at these (absolute $GITHUB_WORKSPACE path — the cc shim cd's, so relative wouldn't resolve); macOS keeps Homebrew clang direct. - cmake + perl in the toolchain action for aws-lc-sys (builds AWS-LC from src). - Fix the indexmap local fixup (version-gate buildscript.run so third-party's 1.x keeps has_std while 2.x/wasm stays no_std). Re-buckified third-party. Verified: node binary still builds on macOS. Assisted-by: Claude:claude-opus-4.8
Run #8: openssl cleared (the OPENSSL_LIB_DIR fix worked). But secp256k1's wasm.c still failed — the [wasm] cc wrapper was at $GITHUB_WORKSPACE (the driver's path), and rebuck2 dispatched the compile to a worker where the buck-baked absolute path need not resolve. Install wasm-cc.sh/wasm-ar.sh to /usr/local/bin in the composite (runs on every runner) and point [wasm] cc/cxx/ar there — a uniform path. (aws-lc-sys still fails separately: its build-script C feature-test link doubles the CRT — duplicate _start/_init under ld.lld — investigating.) Assisted-by: Claude:claude-opus-4.8
secp256k1's wasm.c compile exits 1 with empty stderr — buck's buildscript_run swallows the compiler diagnostic, making it impossible to see WHY clang fails (the command itself looks correct: --target=wasm32-unknown-unknown, -mcpu=mvp, -I wasm/wasm-sysroot). Capture clang's stderr in the wrapper and re-emit it on failure so the real error reaches the driver log. Temporary diagnostic. Assisted-by: Claude:claude-opus-4.8
buck's buildscript_run swallows the compiler stderr (even the wrapper's re-emitted copy), so 10 runs couldn't see WHY clang fails secp256k1's wasm.c. Add a standalone step: fetch the crate, run the exact compile via wasm-cc.sh, surface clang's real diagnostic. continue-on-error so it never gates the job. Temporary. Assisted-by: Claude:claude-opus-4.8
The crates.io API needs a User-Agent; static.crates.io serves the .crate tarball directly. This lets the diagnostic step actually run clang on secp256k1's wasm.c and surface the real error. Assisted-by: Claude:claude-opus-4.8
Subset flags compiled OK — so it's one of the omitted flags (the ENABLE_MODULE_* incl. ellswift, -Dprintf(...)=, -Wall -Wextra). Use the exact buck flag set to surface the real error. Assisted-by: Claude:claude-opus-4.8
The compile works standalone but fails under buck's cc_shim, which passes --ld-path=<ld_shim> to the compiler. For a -c compile that's an unused arg; cc-rs's -Werror=unused-command-line-argument turns it fatal. Test A (with --ld-path) should fail, B (+ -Qunused-arguments) should pass -> the fix. Assisted-by: Claude:claude-opus-4.8
buck's cc shim passes --ld-path=<ld_shim> and -fno-sanitize=all to the C compiler on every invocation, including the -c compiles cc-rs runs for secp256k1-sys's vendored wasm.c, which never link. clang flags --ld-path as an unused argument; cc-rs builds with -Werror=unused-command-line-argument, so it became a fatal error -- and buck's buildscript_run wrapper swallowed the stderr, leaving an empty diagnostic. Add -Qunused-arguments to wasm-cc.sh so clang tolerates the (genuinely unused, on a compile) link flag. Confirmed in CI: identical flags fail with --ld-path, pass with -Qunused-arguments. Removes the DEBUG repro step now the cause is fixed. Assisted-by: Claude:claude-opus-4.8
aws-lc-sys: force the CMAKE builder on Linux (AWS_LC_SYS_CMAKE_BUILDER=1) with plain CC/CXX. The default cc_builder compiles+links compiler-feature probe executables, and buck's cc shim routes links through --ld-path=<ld_shim> which re-drives with clang++, doubling crt1.o -> 'ld.lld: duplicate symbol _start'. cmake builds a static lib and, with plain clang/ld, avoids the double. Gated to Linux; macOS's ld64 tolerates the double so cc_builder still works there. secp256k1 wasm: -Qunused-arguments didn't clear it and cc-rs eats the compiler stderr. Add a WASM_CC_LOG sink to wasm-cc.sh + a forced-local debug build so the real clang diagnostic survives on the driver. TEMP debug, removed once diagnosed. Assisted-by: Claude:claude-opus-4.8
aws-lc cmake path panicked in the `cmake` crate: 'environment variable PROFILE not defined'. The crate reads Cargo's PROFILE/DEBUG for CMAKE_BUILD_TYPE; reindeer's cargo_env omits them. Set PROFILE=release, DEBUG=false (Linux env). wasm-cc.sh: the secp256k1 wasm compile succeeds under local execution but fails under remote (worker) execution, where the action env may not carry a PATH with /usr/bin. Resolve clang via absolute /usr/bin/clang before falling back to a bare PATH lookup. Assisted-by: Claude:claude-opus-4.8
Bump the external_cell_fixups / shared_fixups pin to buck2-fixups@de3d929, which sets local_only=True on cargo build-script run actions. Non-hermetic build scripts (openssl-sys version probe, secp256k1-sys wasm C compile) run on a worker under distributed RE with a divergent env and fail/misdetect; pinning them to the driver fixes secp256k1 (wasm compile) and openssl (ffi version cfgs). Hermetic rustc/cxx compiles still distribute. Fixups dir is unchanged between the old and new pin, so third-party/BUCK is unaffected. Assisted-by: Claude:claude-opus-4.8
Build scripts probe the host and shell out to cc/cmake reading undeclared absolute system paths; dispatched to a remote worker their env diverges and they misbehave (openssl-sys version probe misdetects -> openssl references absent ffi symbols; secp256k1-sys wasm C compile fails). All succeed run locally. Pre-build the build-script-run closure with --prefer-local (host third-party under linux-x86_64, wasm-deps under wasm32, each matching the main build's config so it cache-hits), then the main distributed build reuses those outputs while the hermetic rustc/cxx compiles still distribute. Achieves the effect of local_only without touching the bundled prelude. Reverts the inert external_cell_fixups pin bump (bundled prelude isn't overlaid by that mechanism) and drops the WASM_CC_LOG debug sink. Assisted-by: Claude:claude-opus-4.8
The build-script-run rule has no default output, so 'buck2 build <target>' built nothing (log: 'does not have any outputs') -- the pre-build was a no-op and the main build ran the scripts remote for the first time. Build the [out_dir] sub-target to actually execute each build script, and use --local-only (hard force) instead of --prefer-local (a hint the hybrid executor ignored when workers were idle). set -f stops the shell globbing the [out_dir] suffix. Assisted-by: Claude:claude-opus-4.8
Building all 58 build scripts --local-only surfaced protoc-needing ones (sc-network, litep2p) that build fine distributed but panic locally without protoc. Restrict the local pre-build to the two crates whose build scripts are non-hermetic in a way RE breaks: openssl(-sys) host version probe and secp256k1-sys vendored wasm C. Also the decisive test of whether the main distributed build reuses the pre-built outputs. Assisted-by: Claude:claude-opus-4.8
The wasm compile fails under buck on Linux (driver + worker), not only remote; cc-rs eats clang's stderr. The --local-only pre-build reproduces it on the driver's persistent fs, so have wasm-cc.sh append clang's diagnostic to /tmp/wasm-cc-fail.log and dump it in an always() step. TEMP debug. Assisted-by: Claude:claude-opus-4.8
/tmp is sandboxed by rebuck2's local executor so the diagnostic never reached the host. Write it into the action CWD (under buck-out, the real materialization root) and grep buck-out for it in the dump step. TEMP debug. Assisted-by: Claude:claude-opus-4.8
The secp256k1 wasm compile error can't be surfaced in CI (cc-rs captures the compiler stderr and rebuck2's executor tears down the action fs before a dump step can read any side file). Drop the local pre-build + diagnostic scaffolding and keep the straightforward distributed build; the real fixes (-Qunused-args and absolute clang in wasm-cc.sh, aws-lc cmake+PROFILE) stay. Remaining blockers are documented for a local-Linux repro. Assisted-by: Claude:claude-opus-4.8
…kers Assisted-by: Claude:claude-opus-4.8
Verified end-to-end in an ubuntu:24.04 container (clang 18.1.3, matching CI). Five root causes, all previously hidden because cc-rs swallows compiler stderr: - secp256k1 wasm: the wasm cc-shim exec'd the macOS Homebrew clang path (absent on Linux -> FileNotFoundError). The [wasm] cc override in .buckconfig.local is root-cell-scoped, but toolchains/BUCK reads it from the toolchains cell. Pass it as a global -c wasm.cc/cxx/ar (BUCK_WASM_CFG) which reaches every cell. - openssl: reindeer bakes openssl-sys's DEP_OPENSSL_* at buckify time from the host (macOS 3.5) -> openssl-0.10 gated 3.3+ symbols the ubuntu 3.0.13 ffi lacks. Pin DEP_OPENSSL_VERSION_NUMBER=30000000 + restate DEP_OPENSSL_CONF (fixup env replaces, not merges) via third-party/fixups/openssl. - ittapi-sys: a shared fixup set buildscript.run=false, so jitprofiling.c (iJIT_* symbols) never compiled and the node link failed undefined. Local fixup re-enables it (wasmtime profiling pulls ittapi on Linux). - protoc: composite installs protobuf-compiler (sc-network*/litep2p prost-build). - binaryen: composite installs pinned v130 (ubuntu apt v108 lacks --signext-lowering, which the runtime-wasm blob genrule needs). macOS build unaffected (verified): ittapi/openssl paths are Linux-only. Assisted-by: Claude:claude-opus-4.8
The build is correct (full node+toolkit+wasm blob verified green in an ubuntu:24.04 container), but the distributed CI driver got OOM-reclaimed mid- build (exit 143, shutdown signal) at ~83% done: it runs the buck2 daemon + a co-worker + all local_only actions (build scripts, metadata gen, final links) on a 2-core/7GB free runner. 10G swap absorbs the peak. Assisted-by: Claude:claude-opus-4.8
fallocate on /swapfile failed 'Text file busy' — GitHub runners ship an active 4G /swapfile. Add ours on /mnt (the large ephemeral disk) instead. Assisted-by: Claude:claude-opus-4.8
…tion)
The driver kept dying mid-build with exit 143 ("runner received a shutdown
signal"), not a build error, and swap didn't help — it's disk, not memory. The
full build's buck-out is tens of GB (59G seen locally) and the rebuck2 CAS store
adds more; both default onto the free runner's ~14G / and overflow it, so the VM
is reclaimed. Symlink buck-out to /mnt (~70G) and set the driver store input to
/mnt/rebuck2. Keep a 6G swap for local rustc peaks.
Assisted-by: Claude:claude-opus-4.8
…he fix) The runner already has an active /mnt/swapfile, so fallocate failed 'Text file busy' and skipped the whole step incl. the buck-out redirect. Swap isn't the fix anyway (disk is), so make it best-effort on a fresh path and let the buck-out + store relocation to /mnt always run. Assisted-by: Claude:claude-opus-4.8
The symlinked buck-out let the build run out of the disk trap (4739 commands,
no more exit 143) but buck2's materializer resolved the symlink inconsistently
under rebuck2 — sources landed where local rustc couldn't read them ('couldn't
read src/lib.rs'). A bind mount is transparent: buck2 sees a real buck-out dir
backed by /mnt's 70G.
Assisted-by: Claude:claude-opus-4.8
Assisted-by: Claude:claude-opus-4.8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds an experimental buck2 build of midnight-node (parallel to the existing
cargo and bazel builds) that culminates in a distributed CI workflow: instead
of paying for 16-core machines, one
driverjob owns thebuck2invocation andN free
ubuntu-latestworkerjobs lend their cores over agilescope/rebuck2
iroh P2P mesh. Compile/link/test actions dispatch to whichever runner is free;
blobs move peer-to-peer; rendezvous is keyless via
github.run_id.Full design write-up:
docs/buck2-notes.md.Build system
scripts/buck2/*).crate universe (
wasm-deps/) + first-party wasm variants viaselect()oncpu:wasm32, cdylib →wasm-opt→ zstd (tools/wasm-compress) →include_bytes!inject. 84 substrate exports; the node builds a dev chainspecfrom it.
buck2 test root//... --exclude ci-infra→ Pass 1460 / Fail 0 locally.Distributed CI (
.github/workflows/buck2-distributed.yml)re-execplatform (platforms/defs.bzl).toolchains/BUCKreads[wasm] cc/cxx/arfrom config(macOS Homebrew LLVM defaults; CI overrides with distro clang),
wasm-lld.shlocates rust-lld via the sysroot.
.github/actions/buck2-linux-toolchaininstalls rustc +
wasm32v1-none, clang/lld, wasm-opt.ci-infralabel and are excluded.🗹 TODO before merging
git rebase --signoff.install-buck2latest) and wiring store persistence.📌 Submission Checklist
git commit -s) for the DCOdocs/buck2-notes.mddocs/buck2-notes.md)🧪 Testing Evidence
Locally (macOS arm64):
buck2 build root//node:midnight-node-bin→ builds, links,--version→midnight-node 2.0.0.buck2 build root//:runtime-wasm→ 1.1 MB blob, substratesp-maybe-compressed-blobformat (magic prefix + zstd), 84 exports.buck2 test root//... --exclude ci-infra→ Pass 1460, Fail 0.actionlintclean; rebuck2 action inputs verified against theiraction.yml.Not yet run: the distributed CI itself (needs this push). The local Docker daemon
and the x86 test box were unavailable, so the Linux build is validated on the
first CI run; the toolchain action fail-fasts if a runner's clang lacks the wasm
backend.
🔱 Fork Strategy
Links
Distributed builder: gilescope/rebuck#2