Commit 3fec883
feat(build): scope the dependency cache per package, and make hits actually skip work (2026.7.30.2) (#317)
* docs: dep build-cache scoping design (global cache is currently a no-op)
* docs: add E8 (fast-path delivers wrong profile) + E9 (GCC BMI CRC binding), justify the recursive F axis
* feat(build): scope the dependency cache per package, and make hits actually skip work (2026.7.30.2)
The global dependency cache existed but its net benefit was zero, for two
independent reasons.
It never hit across projects. The key was the whole-project fingerprint, whose
flags field serializes every package in the graph INCLUDING the root — its name,
its version, its [build] flags. So bumping a project's own version invalidated
every dependency it had, and two projects with identical dependencies and
toolchain shared nothing. Measured on one machine: 26 GB across 1198 fingerprint
directories, compat.zlib@1.3.2 stored 162 times, 15 distinct std module
identities occupying 1014 directories (16.1 GB where ~0.5 GB was needed).
And when it did hit, nothing was saved. Artifacts were copied into the build dir
from inside prepare_build while those paths stayed declared as compile edge
outputs — and ninja treats an output it has no command line for in .ninja_log as
dirty, which a fresh build dir always is. Every "cached" unit was recompiled
while the CLI printed "Cached".
ninja explain: command line not found in log for obj/zutil.o
ninja explain: obj/zutil.o is dirty
Keying is now per package (new mcpp.build.cache_key): toolchain identity,
language/dialect, profile, package identity, the package's own build config, and
— recursively — the keys of its direct dependencies. Nothing about the consumer,
which is sound because the root's [build] flags verifiably do not reach
dependency translation units. The recursion is not conservatism: GCC embeds a
CRC of an imported module's BMI into the importer's BMI, so an importer's
artifacts are bound to the exact upstream artifacts they read.
Hits now emit stage_file edges instead of compile edges (and no scan/dyndep
edges), so ninja has a command-line record for the staged outputs. Artifacts
land where a compile edge would have put them, leaving link edges, BMI implicit
inputs and runtime deployment unchanged. The status line carries the unit count
it saved, because a bare "Cached" was printed for months while every unit was
recompiled behind it.
Three correctness defects had to land with it, all of them harmless only while
the cache was a no-op:
- The profile was not an invalidation axis. --dev, --release and --profile dist
shared one fingerprint, one build dir and one cache entry, so a release build
would have been served -O0 -g objects.
- .build_cache keyed fast-path entries by target triple alone, and the fast path
only refuses to run for an EXPLICIT profile flag. `mcpp build --release`
followed by a bare `mcpp build` reported success in 0.00s and left the release
artifacts in place. This one was live regardless of the cache.
- Transitively reached path/git dependencies were cached: the exclusion
predicate consulted the root manifest's dependency maps, where a transitive
package does not appear. Their sources can change without name@version
changing.
Also: --cache=global|local|off (with --no-cache as a deprecated alias for off,
and its inaccurate help text corrected), and mcpp cache grown into something
operable — dir / gc with a real LRU / clean --deps|--std|--all|--legacy /
list --json / verify, with each entry now describing itself in entry.json so a
suspected wrong hit can be audited.
The cache root is $MCPP_HOME/build-cache/v1, deliberately not $MCPP_HOME/cache:
that name belongs to the index metadata cache, whose reset path removes the
whole directory.
Design: .agents/docs/2026-07-30-dep-build-cache-scoping-design.md
Plan: .agents/docs/2026-07-30-dep-build-cache-implementation-plan.md
* fix(build): refuse to cache an index package whose upstream is local
A key covers an upstream package by folding in that package's KEY, and a local
package's key covers its file list but not its file contents — nothing could,
without hashing a tree that may change between the hash and the compile. So a
cached downstream entry would keep looking valid after a local upstream's source
was edited.
No index descriptor can declare a path dependency today, which makes the shape
unreachable in practice. Enforced structurally anyway: "unreachable today" is
exactly how the transitive path-dep leak got in.
Also asserts the invariant that cache-served units keep their
compile_commands.json entries — they stay in the plan on purpose so clangd does
not lose the dependency's sources.
* fix(build): sequence staged artifacts before compilation, and stop hiding std entries
Three fixes from the first CI round.
Module partitions. Replacing a package's compile edges with stage edges also
removes the ordering those compile edges carried. A consumer that imports `pkg`
has `pkg`'s BMI in its dyndep and nothing else — the partition BMI `pkg:part` was
reached only because `pkg`'s own compile edge depended on it. With independent
stage edges ninja may start the consumer while the partition is still unstaged:
error: failed to find module file for module 'mcpplibs.cmdline:options'
macOS CI hit it; Linux won the race, which is why it is now an invariant rather
than a scheduling accident. Every staged artifact becomes an ORDER-ONLY
prerequisite of every non-staged edge, aggregated through one phony so no edge
repeats the list (mcpp#274: long ninja lines are how a 50781-character command
line blew past cmd.exe's 8191 limit). Order-only is the right strength — the real
content dependencies are still declared where they always were, so a changed BMI
still invalidates its consumers; this adds sequencing, not dirtiness. Verified
locally on both GCC and Clang against mcpplibs.cmdline, which has a `:options`
partition.
Ages were computed against the wrong epoch. file_time_type is
std::chrono::file_clock, whose epoch is not the Unix epoch, so `cache list`
printed "74509d ago". Converted through clock_cast.
Test fallout, all of it real:
- 22_doctor_cache_publish asserted `cache list` was empty after `mcpp self
doctor`, which precompiles a std module. That assertion only held because the
old `cache list` walked dep entries and skipped std ones — and hiding them is
how 16 GB of duplicated std BMIs went unnoticed. The empty-cache check moved
ahead of doctor; the std entry is now asserted to be visible, with a bound on
the age column that would have caught the epoch bug.
- 40_llvm_bmi_cache used `--no-cache` to force a cold first build, then expected
the second build to reuse it. `--no-cache` is now an alias for `--cache=off`,
which means neither read NOR write — so it left nothing to reuse. The test's
actual intent (fresh MCPP_HOME is already cold) is now what it says, and it
additionally asserts zero compile edges and the partition sequencing. The
semantic tightening is called out in the CHANGELOG: a mode named `off` that
still writes the cache would not be defensible.
- 98_reflection_import_std grepped $MCPP_HOME/bmi, the pre-v1 root. It passed
locally purely because this machine still holds 26 GB of legacy entries, one of
which happened to record -freflection — a false green of exactly the kind the
plan warned about. It and two siblings now select the std entry by CONTENT
instead of `find | head -1`, which is no longer well-defined: one MCPP_HOME can
hold several std identities now, and that is the point.
* fix(cache): compute entry age without assuming file_clock shares an epoch
libc++ does not provide std::chrono::clock_cast, so the previous fix broke every
Clang build. Measure the mtime as an offset from now IN THE FILE CLOCK and apply
that offset to the system clock instead: no shared epoch, no conversion trait,
and "how long ago" is all any caller wants. Verified compiling and behaving
under llvm@22.1.8 + libc++, which is the configuration that broke.
* fix(build): attribute a source to its most specific package root
Package roots can nest — a workspace member lives under the workspace root — and
taking the first matching root filed the member's sources under the outer
package, i.e. into the wrong cache key. Index payloads live in the xpkgs store
and cannot be shadowed this way, so no cached entry is affected today; resolving
by specificity rather than by iteration order is what keeps that true if roots
ever move.
* fix(cache): scope gc's summary figure to package entries
gc deliberately never evicts std entries — one is shared by every project on the
machine and costs ~30 s to rebuild, so trading it for a little disk is the wrong
trade. But its summary reported the remaining PACKAGE bytes as the cache size, so
a run that freed everything in scope printed "cache now 0.0 B" with tens of MB of
std BMIs sitting right next to it. e2e now pins both halves: the budget is met,
std survives, and the figure says what it measured.
* docs: note the cache-root deviation in the design doc itself
A reader hits the design before the plan, and the design's layout section still
said $MCPP_HOME/cache/v1 — a path that would nest the build cache inside the
index metadata cache, whose reset removes the whole directory.
* fix(build): make the fast path honour the declared cache mode
Same defect shape as the profile one this PR already fixes, in the switch this PR
adds. A build.ninja generated under `--cache=global` contains stage_file edges
reading the global cache. `.build_cache` did not record the mode, and a bare
`mcpp build` (no CLI flag, so nothing bypasses the fast path) replayed that graph
even when the manifest said `cache = "local"` — using the cache the project just
declared it did not want. Ruling the cache out is `local`'s entire purpose, so
this defeated the feature silently.
Reproduced with a warm cache and an untouched manifest, which is what makes it
reachable: editing mcpp.toml would have invalidated the fast path on mtime, so
only the recorded mode can settle it.
build --cache=global → 15 stage edges
build → 15 stage edges [build] cache = "local" ignored
after → 0 stage edges
Mode selection moves into resolve_cache_mode, a pure function of (manifest,
override, environment), so prepare_build and both fast paths settle it from one
rule — the same treatment resolve_profile_name got. prepare_build keeps sole
ownership of the diagnostics: an unparseable value must be reported once, by the
invocation that actually resolves the build, and must fall through to the next
source rather than silently meaning "global".
`.build_cache` records the mode alongside the profile; a mismatch is a miss, and
entries predating the field have an empty mode that matches nothing. No
migration, self-healing on first rebuild.
* docs: record the fast-path graph-identity gap as a follow-up
The fast path replays build.ninja on the premise that this request would generate
the same graph. This PR pins two axes (profile, cache mode) into .build_cache, but
MACOSX_DEPLOYMENT_TARGET, MCPP_VERIFY_MODGRAPH and MCPP_SCANNER also change the
graph and are recorded nowhere — all three predate this PR, and closing them needs
a cheap graph identity rather than re-deriving the fingerprint, which is exactly
what the fast path exists to avoid. Written down instead of half-fixed.
---------
Co-authored-by: sunrisepeak <speakshen@163.com>1 parent b794856 commit 3fec883
41 files changed
Lines changed: 4879 additions & 640 deletions
File tree
- .agents/docs
- docs
- zh
- src
- bmi_cache
- build
- cli
- manifest
- toolchain
- tests
- e2e
- unit
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 419 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 622 additions & 0 deletions
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
6 | 74 | | |
7 | 75 | | |
8 | 76 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
| 149 | + | |
149 | 150 | | |
150 | 151 | | |
151 | 152 | | |
| |||
717 | 718 | | |
718 | 719 | | |
719 | 720 | | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
720 | 726 | | |
721 | | - | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
722 | 771 | | |
723 | 772 | | |
724 | 773 | | |
| |||
744 | 793 | | |
745 | 794 | | |
746 | 795 | | |
747 | | - | |
| 796 | + | |
748 | 797 | | |
749 | 798 | | |
750 | 799 | | |
| |||
756 | 805 | | |
757 | 806 | | |
758 | 807 | | |
759 | | - | |
| 808 | + | |
760 | 809 | | |
761 | 810 | | |
762 | 811 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
139 | 139 | | |
140 | 140 | | |
141 | 141 | | |
| 142 | + | |
142 | 143 | | |
143 | 144 | | |
144 | 145 | | |
| |||
506 | 507 | | |
507 | 508 | | |
508 | 509 | | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
509 | 513 | | |
510 | | - | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
511 | 553 | | |
512 | 554 | | |
513 | 555 | | |
| |||
529 | 571 | | |
530 | 572 | | |
531 | 573 | | |
532 | | - | |
| 574 | + | |
533 | 575 | | |
534 | 576 | | |
535 | 577 | | |
| |||
540 | 582 | | |
541 | 583 | | |
542 | 584 | | |
543 | | - | |
| 585 | + | |
544 | 586 | | |
545 | 587 | | |
546 | 588 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
0 commit comments