Adding more fast tier benchmarks by splitting full projects by different root contracts - #15
Adding more fast tier benchmarks by splitting full projects by different root contracts #15rodiazet wants to merge 1 commit into
fast tier benchmarks by splitting full projects by different root contracts #15Conversation
rodiazet
changed the title
fast tier benchmarks by splitting full projects by different root contracts
blishko
left a comment
blishko
left a comment
There was a problem hiding this comment.
So, the idea is that if you select the parent directory, it will also run benchmarks in subdirectories? Is that a good idea?
|
Also, can someone explain to me how are tags decided? Why some benchmarks have tags like |
Not exactly. It runs compilation benchmarks on the import closures. We want to have more benchmarks in the fast tier but most of the project are too big to get into this tier. So the script divides the project to submodules together with the imports they use and then based on the compilation time they need you can assign to a proper tier. I assumed for now that the contract in the fast tier compiles in <5sec. |
|
ce6ce81 to
fdbed9a
Compare
7784b4d to
97795c4
Compare
fdbed9a to
767033c
Compare
91f2153 to
a48ace0
Compare
767033c to
231e20c
Compare
|
Closed by mistake |
clonker
left a comment
clonker
left a comment
There was a problem hiding this comment.
Generally a good thing to have! But I think the implicit globbing is a bit weird. I'd rather have it explicit. What do you think?
| benchmark_dir = Path(benchmark_dir) | ||
| benchmarks = _load_benchmarks_file(benchmark_dir / "benchmarks.toml") | ||
|
|
||
| for sub_toml in sorted(benchmark_dir.glob("*/benchmarks.toml")): |
There was a problem hiding this comment.
This merges whatever it finds. That's also the exact same namespace gas.py:ensure_project clones Forge projects into. I'd prefer an explicit include = ["subsets"] in the top-level benchmarks.toml I think.
| import re | ||
| import sys | ||
|
|
||
| IMPORT_RE = re.compile(r'import\s+(?:{[^}]*}\s+from\s+)?["\']([^"\']+)["\']') |
There was a problem hiding this comment.
This doesn't match import * as XY from "path";, does it?
| if cur in seen: | ||
| continue | ||
| if cur not in sources: | ||
| print(f"warning: {cur} not in sources, skipping", file=sys.stderr) |
| root_path = benchmark_dir / f"{name}.json" | ||
| if root_path.is_file(): | ||
| return root_path | ||
| for sub_toml in benchmark_dir.glob("*/benchmarks.toml"): |
| config.get("source"), | ||
| config.get("version"), | ||
| benchmark_dir, | ||
| input_file.parent, |
There was a problem hiding this comment.
this is just the benchmark_dir, isn't it?
231e20c to
66c6b13
Compare
66c6b13 to
666277e
Compare
3 participants
This PR introduce a mechanism which split chosen existing projects into smaller parts covering different parts of the compiler pipeline.
They are generated by a dedicated
subset.pyscript into a separated subfolder inbenchmark_data.Depends on #14