ci(benchmark): fetch corpus from published repo (+ via-IR prep) - #527
Draft
nebasuke wants to merge 1 commit into
Draft
ci(benchmark): fetch corpus from published repo (+ via-IR prep)#527nebasuke wants to merge 1 commit into
nebasuke wants to merge 1 commit into
Conversation
nebasuke
added
the
ci:compile-benchmark
Stop vendoring the standard-JSON fixtures in tests/benchmark/fixtures and fetch them from the hardhat-published corpus repo (nomic-foundation-automation/hardhat-benchmark-results) at a pinned commit, via a non-cone sparse checkout that materializes only the benchmarked files. This is the long-term source the README already anticipated; the pinned commit is now the fixtures' provenance (the manifest lives in the corpus repo). Currently the temporary solx-corpus-preview branch — repoint `ref` to the final solx-corpus/ on main once published. Prep the via-IR pipeline as a first-class, opt-in dimension: - run.sh gains a repeatable --variant (default solx-legacy-dwarf) that selects <scenario>/<variant>.json per scenario. The pipeline is encoded in the standard JSON, so no solx flag is needed. --fixtures is now required (the vendored default is gone) and an empty resolution fails loudly. - The checkout fetches both the legacy and via-IR variants of the three scenarios (aave-v4 stays out — it needs EVM_DISABLE_MEMORY_SAFE_ASM_CHECK=1, which the validation pass rejects). via-IR is off by default; a new include_via_ir workflow_dispatch input turns it on. - report.py drops the hardcoded "EVMLA pipeline" title, since via-IR sections can now appear (each section header already carries the variant).
nebasuke
force-pushed
the
compile-benchmark-corpus
branch
from
July 10, 2026 12:11
271de69 to
ef7b8d3
Compare
Compile-time benchmark (
|
| binary | mean ± σ | min … max | vs baseline |
|---|---|---|---|
| pr | 5.006 s ± 0.012 s | 4.994 s … 5.017 s | 1.001× |
| main | 5.003 s ± 0.036 s | 4.978 s … 5.044 s | 1.000× |
| release | 6.622 s ± 0.093 s | 6.556 s … 6.729 s | 1.324× |
openzeppelin-contracts-0.34--solx-legacy-dwarf
| binary | mean ± σ | min … max | vs baseline |
|---|---|---|---|
| pr | 51.716 s ± 0.101 s | 51.634 s … 51.829 s | 0.999× |
| main | 51.775 s ± 0.138 s | 51.673 s … 51.932 s | 1.000× |
| release | 246.753 s ± 4.883 s | 243.754 s … 252.387 s | 4.766× |
uniswap-v4-core-solx--solx-legacy-dwarf
| binary | mean ± σ | min … max | vs baseline |
|---|---|---|---|
| pr | 21.468 s ± 0.062 s | 21.417 s … 21.537 s | 1.000× |
| main | 21.470 s ± 0.057 s | 21.422 s … 21.534 s | 1.000× |
| release | 71.976 s ± 3.864 s | 67.588 s … 74.871 s | 3.352× |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR will stay in draft for a while, until NomicFoundation/hardhat#8415 is slowly merged into Hardhat (in pieces). This is to prove the concept works, and the approach can be continued in HH3.
Claude summary
Follow-up to #518 (now merged). Two changes to the compile-time benchmark:
Corpus: fetched, not vendored
The fixtures leave the repo (
tests/benchmark/fixtures/and itsmanifest.jsonare deleted). A new Checkout benchmark corpus step pulls them from the hardhat-published corpus reponomic-foundation-automation/hardhat-benchmark-results, via a non-cone sparse checkout that materializes only the benchmarked files (not the whole corpus). The pinned commit is now the fixtures' provenance — the manifest lives in the corpus repo.via-IR: prepped, off by default
run.shgains a repeatable--variant(defaultsolx-legacy-dwarf) that selects<scenario>/<variant>.jsonper scenario. The compiler pipeline is encoded in the standard JSON, so switching variants needs no solx flag.include_via_irworkflow_dispatchinput turns it on.workflow_dispatchruns upload the artifact but don't post the PR comment (the comment step is PR-gated). A dedicated label is the path if via-IR-with-comment on a PR is wanted later; kept to the dispatch input here to avoid extra label-gating machinery.report.pydrops the hardcoded "EVMLA pipeline" title, since via-IR sections can now appear (each section header already carries the variant).Fixtures
Same three scenarios timed as before (EVMLA/legacy-DWARF):
ens-verifiable-factory-solx(52),openzeppelin-contracts-0.34(422),uniswap-v4-core-solx(157).aave-v4is never fetched — its dumps requireEVM_DISABLE_MEMORY_SAFE_ASM_CHECK=1, which isn't part of standard JSON, sorun.sh's validation pass would abort on it.Also hardened:
--fixturesis now required (the vendored default is gone), and an empty variant resolution fails loudly instead of feeding hyperfine a literal glob.Validation
solx/hyperfineagainst a fake corpus: both guards fire (missing--fixtures; unknown variant); default → 2 legacy fixtures;--variant solx-legacy-dwarf --variant solx-via-ir-dwarf→ 4 fixtures with correct<scenario>--<variant>naming;report.pyrenders all sections with the generalized title.bash -n run.shclean.actionlintclean apart from the pre-existingsolx-linux-amd64custom-runner-label false positive (from the temporary-runner commit on the base branch).