perf(test-harness): clone and install each project once per run - #571
perf(test-harness): clone and install each project once per run#571nebasuke wants to merge 2 commits into
Conversation
🧪 Integration tests — full matrix · PR vs
|
| Suite | New failures | Size Δ | Gas Δ | Report |
|---|---|---|---|---|
| solx-tester | ✅ 0 | ✅ 0 of 49,512 | ✅ 0 of 63,724 | solx-tester-report.xlsx ↓ |
| Foundry · 25 proj | ✅ 0 (685 pre-existing) | ✅ 0 of 4,972 | ⚪ jitter 24 of 5,193, median <0.1% (not gated) | foundry-report.xlsx ↓ |
| Hardhat · 3 proj | ✅ 0 (40 pre-existing) | ⚪ not collected | ⚪ not collected | hardhat-report.xlsx ↓ |
Compile time — wall-clock tripwire, positive = PR slower (authoritative Δ in ci:compile-benchmark)
| Suite | legacy (agg / median) | viaIR (agg / median) |
|---|---|---|
| Foundry · 25 proj | +0.4% / +0.4% | +0.2% / +0.1% |
| Hardhat · 3 proj | -0.5% / +0.1% | -0.3% / +0.1% |
Within noise — no suite ≥ 5%, no project ≥ 15%.
Bytecode size — PR vs baselines (positive = PR larger; contracts built by both only)
| Suite | Pipeline | vs solc | vs released solx |
|---|---|---|---|
| Foundry · 25 proj | legacy | -13.0% | -0.0% |
| Foundry · 25 proj | viaIR | -1.6% | +0.0% |
nebasuke
added
the
ci:integration-benchmark-full
The Foundry and Hardhat runners re-cloned and re-installed every project for each (compiler x codegen) pair: 4x per project on the PR matrix, 8x in full mode. The only toolchain-dependent inputs are the pragma and config-version seds, whose patterns match their own output, so the checkout is reused and only the seds re-run per toolchain. Three guards replace what the fresh clone provided implicitly: - the sed glob skips node_modules, keeping dependency pragmas untouched now that installs precede the per-toolchain sed; - the Hardhat config file is restored from a pristine snapshot before each sed, since that sed consumes the committed literal version; - Hardhat's cache and artifacts directories and any stale junit report are deleted per toolchain, as its cache cannot distinguish two solx binaries reporting the same base solc version, and a leftover report would mask a test run that produced none.
nebasuke
removed
ci:integration
2b8928a to
280a0e6
Compare
…uards The clone-once reuse re-established per-toolchain isolation piecemeal: a config snapshot/restore, a cache+artifacts purge, and a junit-report deletion on the Hardhat side, and nothing on the Foundry side — where forge's persisted invariant counterexamples (cache/invariant) leaked into the next toolchain's run and could distort the failure-count gate in either direction. Each project checkout is now committed once after setup, and every toolchain iteration starts from that commit via a git reset: checkout of tracked files (submodules included) plus clean of untracked and ignored outputs, keeping node_modules. One primitive covers all per-toolchain state — caches, artifacts, reports, persisted fuzz failures, sed'ed sources and configs — and newly discovered stateful outputs are cleaned without needing their own deletion entry. The seds no longer need to match their own output. Also: - the pragma sed enumerates `git ls-files --recurse-submodules` instead of glob-walking the entire tree per toolchain, which traversed node_modules only to filter it out afterwards; - yarn is installed globally once per run instead of once per requires_yarn project.
|
Need to be careful with these! At first, I used to clone everything just once, but then realized that those projects were finding ways to keep artifacts where I could not find them and it was caching stuff ruining benchmark results in very subtle ways. I'm sure it can be done, but I didn't risk back then as the network was probably faster so nuking the whole state every time was safer. |
Fully agreed. I think I already discovered Hardhat doesn't know the difference between solx/solc when they're both using the same Solidity version, and there's other subtleties. This one will likely be draft for a little while until I either have a clear story, or a very obvious optimisation that doesn't implicitly break stuff. |
2 participants
Stacked on #570, whose 3-line invariant-cap commit rides along in this diff and drops out when it merges.
Claude Summary
The Foundry and Hardhat runners re-clone and re-install every project for each (compiler × codegen) pair — 4× per project on the PR matrix, 8× in full mode. For the Foundry suite that is dominated by repeated
git clone --recursiveand yarn installs; for Hardhat it is repeated npm/pnpm/bun installs. This PR hoists clone, package installs, andforge config --fixto once per project and reuses the checkout across toolchains.The only toolchain-dependent inputs to the sources are the pragma sed and the config-version sed, and their patterns match their own output — so no checkout reset is needed; the seds simply re-run per toolchain with each compiler's version.
What replaces the fresh clone's implicit guarantees
**/*.solglob skipsnode_modules/paths — same end state as before, where the install always followed the sed.0.8.30, which is gone after the first pass, so the config file is restored from a pristine snapshot before each toolchain's sed (also correct under per-compilersolidity_versionoverrides).cache/andartifacts/are deleted before every compile — Hardhat's cache cannot distinguish two solx binaries reporting the same base solc version, and would otherwise reuse the previous toolchain's artifacts. (Foundry already passes--forceon the first build.)junit-report.jsonis deleted before each test run; previously the fresh clone guaranteed its absence, making a test run that produced no report fail loudly.The compiler-identity shim from #529 remains the provenance backstop for any residual staleness.
Measured effect
Standard tier — this PR's
ci:integrationrun (29658942544) vs #570's run (29657293509) — the exact stacked baseline: same day, same standard tier, invariant cap in both. Foundry buckets from the harness status-line timestamps:Foundry step 27.3 → 21.5 min, Hardhat 7.4 → 6.7 min, job total 38.7 → 32.1 min (−6.6 min). The residual 1.3 min of cloning is the irreducible once-per-project fetch.
Full tier (
ci:integration-benchmark-full) — this PR's full-matrix run (29660324611) vs the pre-#570 full-matrix baseline (29643434263), so the deltas include both this PR and #570's invariant cap:Foundry step 93.6 → 66.9 min, Hardhat 17.3 → 15.5 min, job total 114.8 → 86.4 min (−25%).
Checkout reuse perturbed nothing measurable, on either tier — every deterministic signal is identical to the fresh-clone baseline, including on the full matrix where one clone per project is shared across all four compiler lineages (solc, solx-latest, solx-main, solx-PR): 685 Foundry / 40 Hardhat pre-existing failures, 0 new; bytecode size identical across all 4,972 rows; gas 5,193 rows; solx-tester identical. Combined with #570: default
ci:integration~45 → ~32 min (−29%), full matrix ~115 → ~86 min.Local verification
End-to-end runs with two toolchains (both solc 0.8.34, reference + candidate, 2 codegens each), debug
solx-dev:lil-web3,maple-erc20): one clone per project, all 4 toolchains built/tested, 25 contract entries with size rows, exit 0.lil-web3viaIR shows the same 2 test failures on both identical toolchains — deterministic parity, gate correctly silent.diamonds,requires_yarn): yarn install ran exactly once, all 4 toolchains built.hardhat-v2template, bun): one clone,Compiled 1 Solidity file successfullyon all four passes (proving the cache deletion forces recompilation — without it passes 2–4 report "Nothing to compile"), config restore+sed ran 4×, 0 build/test failures, exit 0.cargo check/clippy/fmtclean onsolx-dev.