ci: enable more build caching#1814
Open
ozgb wants to merge 18 commits into
Open
Conversation
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
`cargo auditable build` injects cargo-auditable as RUSTC_WORKSPACE_WRAPPER, so cargo chains it after the kache RUSTC_WRAPPER as `kache cargo-auditable rustc ...`. kache only enters its rustc-wrapper path when the wrapped binary's basename is rustc-family (rustc*/clippy-driver), so the cargo-auditable basename falls through to its clap CLI and aborts the build with "unrecognized subcommand '.../cargo-auditable'". Rename the cargo-auditable binary to a rustc-prefixed name (so RUSTC_WORKSPACE_WRAPPER, set from current_exe(), resolves to .../rustc-auditable which kache recognises and runs in its double-wrapper mode) and keep a cargo-auditable symlink so the `cargo auditable` subcommand is still found on PATH. cargo-auditable selects its mode from argv[1]/CARGO_AUDITABLE_ORIG_ARGS, not its own name, so the rename is transparent to it. Gated under USE_KACHE. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Gate caching in the Earthfile `build` target on a new global `PROD` arg. With PROD=false (default) the target mounts the full cache set — cargo registry/git, /target, and the content-addressed kache store — for fast dev/CI iteration. With PROD=true it declares no CACHE mounts, bypasses the kache RUSTC_WRAPPER, and compiles with --no-cache, so release artifacts are built from scratch with nothing served from any cache. The main workflow sets --PROD=true for every push to main/release/* and exposes a `prod` workflow_dispatch toggle for manual runs, so pushed release images are hermetic while PR CI stays fully cached. Assisted-by: Claude:claude-4.8-opus Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
CARGO_HOME was unset, so cargo defaulted to /root/.cargo, while every CACHE --id cargo-git/cargo-reg mount targets /usr/local/cargo/*. The mounts overlaid a directory cargo never wrote to, so fetched git DBs (e.g. midnight-ledger) landed in the un-cached /root/.cargo/git and were discarded on every source edit, forcing a re-fetch each build. Set CARGO_HOME=/usr/local/cargo in +prep-no-copy (after all cargo-tool installs, which stay in /root/.cargo/bin on PATH) so all inherited build/check/test targets use the cache mounts. Pin git-fetch-with-cli in $CARGO_HOME/config.toml so it holds regardless of workdir. Also finalize the PROD build gate: - drop the CACHE /target mount from +build (intentionally left uncached; kache caches the compile units) - revert the +build ELSE branch to `cargo auditable build` (embeds the SBOM in shipped binaries; c2893ef made it kache-compatible) And remove 11 no-op runner-host git-fetch-with-cli config steps that preceded earthly invocations: BuildKit containers never see the runner's $HOME/.cargo/config, and git-fetch-with-cli is baked into the CI image. The one load-bearing occurrence (build-indexer-images: cargo runs directly on the runner with a tokenized git insteadOf) is kept. Assisted-by: Claude:claude-opus-4-8[1m] Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
0.8.0 self-heals a corrupt cache index instead of bricking every command (relevant since KACHE_CACHE_DIR=/kache is a CACHE mount shared across concurrent targets) and normalizes rustc-env include paths under OUT_DIR for the Substrate cascade, improving cross-clone cache hits. No changes to the RUSTC_WRAPPER / KACHE_CACHE_DIR / KACHE_LOCAL_ONLY / KACHE_FALLBACK env contract we rely on, and the linux-musl asset naming is unchanged. Verified the download + sha256 + extract flow against v0.8.0 (kache --version -> 0.8.0). Assisted-by: Claude:claude-opus-4-8[1m] Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Set CARGO_HOME=/usr/local/cargo (and prepend $CARGO_HOME/bin to PATH) BEFORE the cargo-tool install in +prep-no-copy, instead of after. The `cargo binstall cargo-auditable` now lands in /usr/local/cargo/bin, so cargo's home (git/registry cache, config.toml, and installed tool binaries) is fully self-consistent under one root, and any future build-time `cargo install` works without further PATH tweaks. cargo/rustc are rustup proxies in /root/.cargo/bin (kept on PATH by the CI image) and are unaffected — CARGO_HOME only relocates cargo's data/bin home, and /usr/local/cargo/bin holds no cargo/rustc proxy, so there is no shadowing. The kache RUSTC_WRAPPER-compat rename resolves cargo-auditable via `command -v`, so it follows the new location transparently. Verified against the real CI image: cargo-auditable installs to /usr/local/cargo/bin, `cargo`/`rustc` still resolve to /root/.cargo/bin, the rustc-auditable rename + symlink work, and `cargo auditable` still resolves as a subcommand. Assisted-by: Claude:claude-opus-4-8[1m] Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
+build and +build-node-only declared no /target CACHE mount, so any
source change forced cargo to recompile the whole workspace from scratch
inside the container. kache served the individual compile units, but
cargo could not skip unchanged crates the way it does on the host, where
target/ persists across builds. Add a per-branch-scoped
`CACHE --id target-${CACHE_KEY} /target` to both (matching +check/+test),
so incremental fingerprinting carries across runs and a leaf-crate edit
rebuilds ~only that crate.
Also switch the artifact extraction from `mv` to `cp`: with /target now
persistent, moving the linked binaries out would evict them from the
cache and force a re-link on the next run even when their inputs are
unchanged.
Safety: /target is scoped per CACHE_KEY, so different branches never
share it (the E0046 rmeta-leak concern the header comment describes), and
it uses Earthly's default `locked` sharing, so concurrent builds on the
same CACHE_KEY serialize rather than clobber each other. The PROD
hermetic build declares no mounts and is unaffected.
Assisted-by: Claude:claude-opus-4-8[1m]
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
BuildKit cache-mount ids are not scoped by platform (moby/buildkit#2598), so every target mounting `target-${CACHE_KEY} /target` (+build, +build-node-only, +test, +test-pallet-fixtures, +build-test-toolkit, +rebuild-sqlx, +planner) shared one native /target/release across arches. On the native per-arch runners used today this is harmless — each arch builds on its own buildkitd with separate mount storage — but it would thrash if a single daemon ever built more than one arch (e.g. emulated multi-platform), where cargo would treat the other arch's artifacts as stale and rebuild each alternation. Suffix the id with -${TARGETARCH} (declaring `ARG TARGETARCH` before each cache line, matching the existing convention) so correctness no longer depends on the daemon topology. Same-arch builds still share the cache, so no caching is lost. Non-/target mounts are left as-is: the cargo git/registry caches hold arch-independent sources, and the kache store is already content-addressed on the full compile inputs (arch included). Assisted-by: Claude:claude-opus-4-8[1m] Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
kache is content-addressed (keys include normalized rustc flags), so
check/debug/release entries coexist without collision. Splitting the
store into per-flavor ids (kache-{check,prep,build}) therefore couldn't
prevent wrong hits — it only prevented identical host compiles
(proc-macros, build scripts, identically-flagged deps) from being reused
across flavors. Collapse the 11 mounts to a single `--id kache`, which is
strictly >= the split for hit rate and never serves a wrong entry.
/target stays scoped by CACHE_KEY+TARGETARCH, since cargo fingerprints
(unlike kache) leak across branches.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> # Conflicts: # Earthfile
Left over from a throwaway commit used to bust caches while testing kache; no functional change. Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
kache (content-addressed rustc build cache) added enough failure surface for the reuse to not be worth it. Drop USE_KACHE/KACHE_VERSION, the install-and-wrap-RUSTC_WRAPPER step in +prep-no-copy, the cargo-auditable rustc-wrapper compat rename, and all kache CACHE mounts. Keep the unrelated fixes made alongside it: CARGO_HOME relocated to /usr/local/cargo so the cargo-git/cargo-reg CACHE mounts actually take effect, cargo tools installed into CARGO_HOME/bin, and /target persisted and scoped by CACHE_KEY+TARGETARCH in +build/+build-node-only. Also keeps the PROD hermetic-build gate (minus its kache-specific bypass). Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
+node-image/+toolkit-image tag and push purely by git tree-hash + version, with no PROD marker, and main.yml's "already exists" dedup check keyed off that same tag. continuous-integration.yml's PR builds always run PROD=false but legitimately publish under that same canonical tag too (its downstream jobs consume node_image_tag_no_dev/toolkit_image_tag) - so a PR build, or a PROD=false manual workflow_dispatch of main.yml, could leave the tag already "existing" for a tree that a later real push (PROD=true) builds for. main.yml would then see the tag exists and skip the hermetic rebuild, silently keeping a cache-influenced build as the official release image. Add a separate marker tag ($VERSION-hermetic-$HASH-$ARCH), pushed only when PROD=true, and point main.yml's dedup check at it instead. All existing tag pushes (latest, canonical, dev, public mirror) are unchanged in every mode, so PR CI's tag consumption is unaffected. Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Cached, non-hermetic builds (PROD=false — PR/merge-group CI) previously shared the clean canonical image tag <version>-<treehash>-<arch> with hermetic release builds (PROD=true — main/release pushes), so a cached build could occupy or mask a release image's tag. This was worked around with a separate -hermetic- marker tag. Bake the build mode into the tag instead: PROD=true publishes the clean canonical tag (and moves latest-<arch>); PROD=false publishes only <version>-dev-<treehash>-<arch> (private and public registries). Because the clean tag can now only ever come from a hermetic build, its existence proves hermeticity — so the -hermetic- marker and the dedup logic keyed off it are removed, and main.yml's dedup checks the clean tag directly. CI consumers (e2e, SBOM, genesis, local-env) resolve the -dev tags via the workflow job outputs, so no consumer changes were needed; the toolkit tag gains the -dev infix the node image already used, and push-combined-manifest now publishes a -dev public multi-arch image. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Taking advantage of the fact that our self-hosted runners share an Earthly buildkit instance, this PR contains tweaks to take more advantage of Earthly caches.
mainbuilds are unaffected - caching is turned off when globalPROD=trueis set.Adds Kache, a new zero-copy content addressed Rust build cache specifically benchmarked against SubstrateSpeedup not worth the addition - removed (relevant issue on the kache repo: bench(substrate): close the cross-clone hit-rate gap — 503 .rmeta misses + 255 uncached executables + 60 dep-info refusals kunobi-ninja/kache#431)CARGO_HOME(no more re-fetching git deps)/target) for nonPRODbuildsPRODbuilds are tagged with-devand used in CIPROD=true- so release builds intentionally don't use the build cacheResults:
node/src/lib.rsdoesn't refetch + rebuild the whole project anymore🗹 TODO before merging
📌 Submission Checklist
git commit -s) for the DCO🧪 Testing Evidence
Please describe any additional testing aside from CI:
🔱 Fork Strategy
Links