docs: update build performance benchmarks (3.9.3)#924
Conversation
Greptile SummaryAutomated benchmark update for v3.9.3, recording a significant native-engine improvement: build time dropped from 15.7 ms/file to 3.3 ms/file (↓79%), driven by dramatic reductions in the AST-nodes, complexity, CFG, and dataflow phases. All delta calculations, 50K extrapolations, and phase-sum cross-checks are internally consistent with the embedded JSON record. Confidence Score: 5/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant CI as GitHub Actions (publish.yml)
participant BM as scripts/benchmark.ts
participant NAT as Native Engine
participant WASM as WASM Engine
participant MD as BUILD-BENCHMARKS.md
participant README as README.md
CI->>BM: workflow_dispatch (v3.9.3)
BM->>NAT: full build + query + incremental
NAT-->>BM: "buildTimeMs=2187, nodes=16938, edges=34396"
BM->>WASM: full build + query + incremental
WASM-->>BM: "buildTimeMs=11590, nodes=17497, edges=35859"
BM->>BM: formatEngineResult() → perFile values
Note over BM: Known bug (#925): perFile uses<br/>WASM file count (727) as denominator<br/>for both engines
BM->>MD: prepend new row + embed JSON record
BM->>README: "update summary table & last-updated date"
Reviews (2): Last reviewed commit: "docs: update build performance benchmark..." | Re-trigger Greptile |
| @@ -5,6 +5,8 @@ Metrics are normalized per file for cross-version comparability. | |||
|
|
|||
| | Version | Engine | Date | Files | Build (ms/file) | Query (ms) | Nodes/file | Edges/file | DB (bytes/file) | | |||
| |---------|--------|------|------:|----------------:|-----------:|-----------:|-----------:|----------------:| | |||
There was a problem hiding this comment.
Native per-file metrics don't reconcile with raw totals
The native perFile.* values in the embedded JSON (lines 326–330) can't be reproduced by dividing the raw totals by 727 files, whereas WASM values are internally consistent:
| Metric | Native raw total | ÷ 727 | Reported perFile | WASM check |
|---|---|---|---|---|
| Build (ms/file) | 2,187 ms | 3.0 | 3.3 | 11590/727 = 15.9 ✓ |
| Nodes/file | 16,938 | 23.3 | 25.4 | 17497/727 = 24.1 ✓ |
| Edges/file | 34,396 | 47.3 | 51.6 | 35859/727 = 49.3 ✓ |
| DB (bytes/file) | 29,335,552 | 40,351 | 43,981 | 27901952/727 = 38,380 ✓ |
The native per-file figures are consistent with total / ~663 as denominator (e.g. 2187 / 3.3 ≈ 663, 16938 / 25.4 ≈ 667), suggesting the benchmark workflow counts ~64 fewer files for the native engine than the 727 it reports. The practical effect: the table shows native having 25.4 nodes/file (more than WASM's 24.1), but the raw totals imply the opposite — 23.3 (native) vs 24.1 (WASM). Worth checking whether the native benchmark run's filesProcessed count diverges from its total files count.
There was a problem hiding this comment.
Good catch. Confirmed the root cause: in scripts/benchmark.ts, formatEngineResult() strips the per-engine files field, and the parent process sets the top-level files from primary.files (WASM). The native engine likely processed ~663 files, so its per-file values are internally consistent with its own file count, but the table misleadingly shows 727 (WASM's count) in the Files column for both engines.
This is a bug in the benchmark infrastructure, not in the benchmark data itself. Created #925 with the follow-up label to track the fix (include files in formatEngineResult and use per-engine file counts in the report renderer).
Automated build benchmark update for 3.9.3 from workflow run #712.