Skip to content

docs: update build performance benchmarks (3.9.3)#924

Merged
carlos-alm merged 2 commits intomainfrom
benchmark/build-v3.9.3-20260413-071945
Apr 14, 2026
Merged

docs: update build performance benchmarks (3.9.3)#924
carlos-alm merged 2 commits intomainfrom
benchmark/build-v3.9.3-20260413-071945

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

Automated build benchmark update for 3.9.3 from workflow run #712.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Apr 13, 2026

Greptile Summary

Automated benchmark update for v3.9.3, recording a significant native-engine improvement: build time dropped from 15.7 ms/file to 3.3 ms/file (↓79%), driven by dramatic reductions in the AST-nodes, complexity, CFG, and dataflow phases. All delta calculations, 50K extrapolations, and phase-sum cross-checks are internally consistent with the embedded JSON record.

Confidence Score: 5/5

  • Safe to merge — documentation-only benchmark update with no code changes.
  • All numeric deltas, 50K extrapolations, and phase sums are internally consistent. The only known data-quality issue (native per-file denominator bug) is pre-existing, already identified in the previous review thread, and tracked in fix: benchmark per-file metrics use wrong file count for native engine #925 — it was not introduced by this PR.
  • No files require special attention.

Important Files Changed

Filename Overview
README.md Updates performance table for v3.9.3 (2026-04-13); native build speed shows a dramatic improvement (15.7 → 3.3 ms/file, ↓79%) and 50K extrapolation updated accordingly — all README values are consistent with reported per-file figures in the benchmark file.
generated/benchmarks/BUILD-BENCHMARKS.md Adds 3.9.3 rows to all history tables and embeds the full JSON benchmark record; percentage deltas, 50K extrapolations, and phase breakdowns are internally consistent. The previously-identified native per-file denominator discrepancy (tracked in #925) is unchanged from prior releases and not introduced by this PR.

Sequence Diagram

sequenceDiagram
    participant CI as GitHub Actions (publish.yml)
    participant BM as scripts/benchmark.ts
    participant NAT as Native Engine
    participant WASM as WASM Engine
    participant MD as BUILD-BENCHMARKS.md
    participant README as README.md

    CI->>BM: workflow_dispatch (v3.9.3)
    BM->>NAT: full build + query + incremental
    NAT-->>BM: "buildTimeMs=2187, nodes=16938, edges=34396"
    BM->>WASM: full build + query + incremental
    WASM-->>BM: "buildTimeMs=11590, nodes=17497, edges=35859"
    BM->>BM: formatEngineResult() → perFile values
    Note over BM: Known bug (#925): perFile uses<br/>WASM file count (727) as denominator<br/>for both engines
    BM->>MD: prepend new row + embed JSON record
    BM->>README: "update summary table & last-updated date"
Loading

Reviews (2): Last reviewed commit: "docs: update build performance benchmark..." | Re-trigger Greptile

@@ -5,6 +5,8 @@ Metrics are normalized per file for cross-version comparability.

| Version | Engine | Date | Files | Build (ms/file) | Query (ms) | Nodes/file | Edges/file | DB (bytes/file) |
|---------|--------|------|------:|----------------:|-----------:|-----------:|-----------:|----------------:|
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Native per-file metrics don't reconcile with raw totals

The native perFile.* values in the embedded JSON (lines 326–330) can't be reproduced by dividing the raw totals by 727 files, whereas WASM values are internally consistent:

Metric Native raw total ÷ 727 Reported perFile WASM check
Build (ms/file) 2,187 ms 3.0 3.3 11590/727 = 15.9 ✓
Nodes/file 16,938 23.3 25.4 17497/727 = 24.1 ✓
Edges/file 34,396 47.3 51.6 35859/727 = 49.3 ✓
DB (bytes/file) 29,335,552 40,351 43,981 27901952/727 = 38,380 ✓

The native per-file figures are consistent with total / ~663 as denominator (e.g. 2187 / 3.3 ≈ 663, 16938 / 25.4 ≈ 667), suggesting the benchmark workflow counts ~64 fewer files for the native engine than the 727 it reports. The practical effect: the table shows native having 25.4 nodes/file (more than WASM's 24.1), but the raw totals imply the opposite — 23.3 (native) vs 24.1 (WASM). Worth checking whether the native benchmark run's filesProcessed count diverges from its total files count.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Confirmed the root cause: in scripts/benchmark.ts, formatEngineResult() strips the per-engine files field, and the parent process sets the top-level files from primary.files (WASM). The native engine likely processed ~663 files, so its per-file values are internally consistent with its own file count, but the table misleadingly shows 727 (WASM's count) in the Files column for both engines.

This is a bug in the benchmark infrastructure, not in the benchmark data itself. Created #925 with the follow-up label to track the fix (include files in formatEngineResult and use per-engine file counts in the report renderer).

@carlos-alm
Copy link
Copy Markdown
Contributor

@greptileai

@carlos-alm carlos-alm merged commit b533ac1 into main Apr 14, 2026
17 checks passed
@carlos-alm carlos-alm deleted the benchmark/build-v3.9.3-20260413-071945 branch April 14, 2026 05:16
@github-actions github-actions bot locked and limited conversation to collaborators Apr 14, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant