Release v0.49.0: reconcile V10 parser and performance train by odvcencio · Pull Request #665 · odvcencio/gotreesitter

odvcencio · 2026-08-09T06:50:27Z

Purpose

This pull request replaces the untagged staged v0.49.0 candidate. It
reconciles the active parser train with main and prepares the actual release.

Net changeset map

The branch has 42 commits. Six leading waypoint commits are empty because their
changes already exist on main. The effective history has 36 content commits.

The net patch changes 92 files. It adds 9,792 lines and removes 714 lines. The
categories below include their tests and evidence code.

Area	Approximate share of additions	Default effect	Primary review files
Measurement and fleet controls	31.2%	Tooling only	`cgo_harness/`, `cmd/retry_profile_cert_child/`, `scripts/`
Merge-event census	18.8%	Build-tag only	`merge_event_census.go`, `cgo_harness/merge_event_census_test.go`
Compact core and C4 bytecode	12.8%	Certified compact paths apply. The C4 corridor stays off by default.	`parsercore_c4_program.go`, `parsercore_c4_vm.go`, `internal/parsercorephase0/`
Lean and registry support	10.7%	Opt-in package	`grammars/lean/`, `grammargen/lean_grammar.go`
Parser and recovery	9.6%	Active under certified retry and recovery conditions	`parser_retry.go`, `parser_recover_c.go`, `glr.go`
Fact extraction API	8.7%	New opt-in public API	`fact_program.go`, `understanding.go`
Synthetic-root replay	6.1%	Active during recovered synthetic-root construction	`parser_result_root_build.go`
Release documentation and CI	2.1%	No parser runtime effect	`CHANGELOG.md`, `README.md`, `.github/workflows/ci.yml`

Measurement and census code accounts for half of the additions. Most new lines
provide evidence, bounds, and reproducible gates.

Functional changes

Parser, recovery, and compact execution

Bound the accepted-error retry ladder by pass, wall, stack, merge, and node limits.
Preserve retry candidates until later rungs consume their runtime evidence.
Prevent retry ranking from releasing its current incumbent through an alias.
Add a temporary 262,144-entry recovery memo tier for measured collision pressure.
Share the bounded recovery memo with generalized LR merge comparisons.
Keep uncommon recovery state in a cold parser sidecar.
Add single-derivation iteration and provenance caching to the compact core.
Add guarded REDUCE_CHAIN and REDUCE_SHIFT C4 instructions.
Hash replay frames and gap cursors.
Memoize replay lexing, advance transitions, and closure results.
Store replay outputs in bounded pages with 4-byte spans.

Fact extraction API

Add the FactKind selection mask for definitions, calls, heritage, and imports.
Add FactSet as the language-neutral combined result.
Add NewFactProgram to compile grammar symbols and field identifiers once.
Store one dense 16-bit instruction for each grammar symbol.
Execute all selected fact operations during one depth-first tree traversal.
Reuse each decoded definition for both definition and heritage output.
Preserve exact output order and values from the four individual extractors.
Reject invalid masks and return no output for a different language value.
Keep ExtractDefinitionSpans, ExtractCalls, ExtractHeritage, and ExtractImports stable.

Grammar, measurement, and release controls

Add an opt-in Lean 4 grammar, scanner, highlights, tags, and corpus tests.
Add randomized benchmark, resident-memory, fleet, and merge-census controls.
Add fact and tags lanes to the combined randomized benchmark set.
Trigger full pull request CI when a draft becomes ready for review.

The C4 corridor requires GTS_C4_CORRIDOR=1. Each fused C4 instruction also
requires its own gate. Default parsing does not execute those instructions.

FactProgram is opt-in. Existing extraction calls and parser behavior remain
unchanged until a caller constructs and uses a program.

Performance evidence

Accepted V10 epoch

Accepted epoch: harness_out/gcp/20260808T202958Z-v10-full-5003ffba.
V10 completed 1,435 of 1,435 measurements across 206 languages.
The epoch recorded no crash, out-of-memory event, or runaway retention.

FactProgram

The combined suite ran 20 explicit shuffle seeds. Each seed used one process,
GOMAXPROCS=1, a 750 millisecond benchmark time, and memory reporting.

Definitions and calls on an existing Go tree:

Time changed from 350.5 microseconds to 101.0 microseconds.
Time improved by 71.19 percent with p < 0.001.
Bytes and allocations stayed equal.

All four fact classes on an existing Go tree:

Time changed from 623.43 microseconds to 98.90 microseconds.
Time improved by 84.14 percent with p < 0.001.
Bytes changed from 99.47 KiB to 97.60 KiB, a 1.88 percent improvement.
Allocations changed from 1,012 to 512, a 49.41 percent improvement.

Parse plus definitions and calls stayed neutral:

Time changed from 7.206 milliseconds to 7.192 milliseconds.
The timing result was neutral with p = 0.947.
Bytes and allocations also stayed neutral.

Parser and replay work

Replay transition caching reduced the hard Elixir target latency by 22.62 percent.
Packed closure paging reduced target bytes per operation by 5.15 percent.
Packed closure paging reduced target allocations by 60.48 percent.
Packed closure latency stayed neutral across 20 balanced revision-order pairs.
The prior combined suite kept all 16 latency rows neutral.
The prior combined latency geomean improved by 1.57 percent.

The exact Swift issue #586 witness used three counterbalanced pairs against
origin/main.

Total allocation bytes improved by 11.08 percent.
Mean wall time improved by 4.49 percent. The sample remains noisy.
Maximum resident memory did not improve.
Every run covered all 104,681 source bytes and produced the same Go tree digest.
The remaining C mismatch stays pinned under issue Swift recovery parity vs the locked C oracle (umbrella) #576.

The current Swift profile assigns 16.17 percent flat CPU time to
cNodeMemoSlot. The witness uses the 6 MiB memo tier and records 41,881
collisions. Cache locality is the next packing target.

Merge-event evidence

The stage M0 census measures 104 constructed sources.

C records 191 successful merges.
Go records 15 successful merges.
The ratio is 0.0785.
No source records a Go over-merge.
Twenty-two sources record a C merge without a Go merge.
Go records 142 one-flat-side refusals.
Go records 92 score refusals.
Go records 59 distinct-shape refusals.

The Foo[int](a) witness supports election at the spanning pop. It does not
support an unproved global shallow merge.

Correctness evidence

Focused Docker fact, import, definition, and retention gates pass.
FactProgram matches every legacy output field on six supported languages.
Selection-mask and language-mismatch guards pass.
The isolated heap-retention gate passed five consecutive Docker runs.
Go vet, formatting, shell syntax, and structural checks pass.
Focused Docker replay tests pass.
Elixir no-error, S-expression, and deep-tree parity pass 25 of 25 cases.
Packed replay records zero capacity skips on the measured target.
The compact T3 C oracle passes after the retry alias fix.
Swift no-error parity passes 25 of 25 cases.
Swift S-expression and deep-tree parity pass 22 of 25 cases.
The three Swift mismatches are known generated-grammar gaps.
Lean parses 3,699 official files and 15,540,447 bytes cleanly.
The packaged Lean blob matches its Go grammar definition.

PR CI run 31304643870 passed exact head 5bbe8be1 with 44 green jobs.
Exact-main CI run 31305235288 passed verified merge e1ce6514 with 44 green
jobs. The governed soak started at 2026-08-09T09:23:07Z. It cannot complete
before 2026-08-11T09:23:07Z.

Issue and pull request linkage

Item	Relationship	State after this pull request
#454	Improves shared parser bounds, memory evidence, and downstream diagnostics.	Keep open for remaining downstream gaps.
#586	Reduces allocation on the locked Swift witness and identifies the next cache target.	Keep open.
#576	Preserves the locked Swift C-oracle mismatch ratchet.	Keep open until structural parity is proven.
#649	Adds a compiled language-neutral extraction base for outline consumers. It does not add hierarchy or owner resolution.	Keep open for the requested outline contract.
#645	Depends on safer merge-time election work measured by the new census.	Keep separate and held.
#664	Its opt-in Lean implementation is included here.	Closed as superseded by this release train.

PR #645 remains a held draft. Its rescue-only arm creates 69 truncations while
fixing six witnesses. This release does not integrate that regression.

Mined follow-on targets

Add outline hierarchy and reliable owner resolution for issue Feature request: language-neutral file outline API #649.
Extend FactProgram only with exact language rules and conformance fixtures.
Split or repack the recovery memo lookup before changing broad tree layout.
Test smaller memo tiers and set layouts against the exact Swift witness.
Advance merge election from the M0 census toward the spanning-pop proof.
Promote C4 instructions only after full correctness and randomized timing receipts.
Keep NaN-boxing experimental for records that are not dynamic numeric unions.
Reuse bounded pages and spans for immutable small result vectors.

Release reconciliation

No v0.49.0 tag or GitHub release exists. v0.48.1 remains the latest
immutable release.

After merge:

Dispatch full hosted CI for the exact main commit.
Require correctness, parity, race, and performance gates.
Complete the governed 48-hour soak.
Record a signed Hyphae release receipt.
Dispatch the protected release workflow from main.
Verify the GitHub release and Go module proxy.

Benchmark method

The standardized runner uses one process for each explicit shuffle seed. It
randomizes benchmark order within each process.

Revision comparisons counterbalance baseline and candidate order. FactProgram
comparisons place both equivalent arms in each shuffled combined run.

Settings:

GOMAXPROCS=1
one count for each process
20 shuffle seeds
750 millisecond benchmark time
memory reporting enabled

- Add cgo_harness/attribution command that maps CPU profile samples to eight mutually exclusive components and writes markdown and JSON receipts - Capture endpoint profiles both the diagnostic lane and the shipped Parser.Parse route across four pinned canonical fixtures - Include interleaved A/A noise floor and cost-per-event joins to contextualize component shares - Implement a minimal pprof protobuf decoder to avoid widening the module dependency footprint - Publish first receipt and methodology documentation under docs/

- Add `recovery` attribution component (gate G5: classifier first) so recovery cost cannot hide in `other` - Reclassify pure no-table-action dispatch as `DiagnosticParserCoreRecovery` to separate locked-C recovery handoff from election pause - Upgrade compact T3 harness from root HasError to full structural parity (tree shape, spans, Missing, HasError, fields) - Record 98-to-20 witness gap in manifest denominator block with verification metadata - Log production/compact vs C oracle divergences without asserting, per S1 stop rule

- Add a unit test to pin the `labeled_statement` and `_property_name` declared-conflict retention rule directly against the generator. - Correct the root-cause comment in the javascript election parity test. The generator already retains the fork, but the shipped blob predates the fix. - Update the changelog to record the corrected root-cause and the new regression pin.

…uage symbols - Add ExternalScannerForLanguage to JavaScriptExternalScanner so it binds token slots positionally to the attached Language's external symbols, matching the pattern used by kotlin/python/swift/dart/rust/hcl scanners - Replace hardcoded jsSym* absolute IDs in Scan and helper functions with a per-instance symbol table resolved from the bound Language - Add jsDefaultSymTable fallback and remapValidSymbols translation for when the Language's external count or order diverges from jsExternalSymbolNames - Add regression test with a synthetic Language that asserts bound symbols resolve to the Language's own numbering, not the pinned defaults - Extend the positional binding umbrella test to pin the shipped javascript.bin's binding as a permanent guard

…age symbols - TypeScript and TSX external scanners used hardcoded symbol IDs (tsSym*, tsxSym*) in SetResultSymbol calls, which silently mistyped every external scan result when a grammargen-regenerated blob shifted the automaton's absolute symbol numbering - Implement ExternalScannerForLanguage on both scanners to bind token slots positionally by external index via bindExternalScannerSymbolNames, matching the pattern already used by JavaScript, Kotlin, Python, Swift, Dart, Rust, and HCL scanners - Add default symbol tables and external symbol name lists so Scan resolves results through the bound table instead of pinned constants - Add regression tests that verify positional binding with synthetic languages whose symbol IDs differ from the pinned defaults - Extend the positional binding table test to pin the currently-shipped TypeScript and TSX bindings as permanent regression guards

…nd fast path - Add dispatchSupported field to ActionRowDescriptor so the generic scheduler dispatch loop can skip the per-pass unsupported-cell check and its token-struct copy for Shift, Reduce, and Conflict rows, which are provably supported from the row kind alone - Extract condenseDirectAppend fast path in condenseWithOutcomeAtomic for the case where no incumbent boundary exists, avoiding the ~200-line fold-comparison block when only one candidate link is present

- Add runbook for v10 full-fleet performance measurement on GCP - Document VM provisioning, corpus authentication, and bisect ladder - Add index file that points to the canonical bytecode stream specification

- Change directory to `cgo_harness` before running the corpus source tool - Update the relative path for `real_corpus_sources` to match the new working directory

- Change free-space requirement from 100 GB before scan to 40 GB after materializing the corpus - Add instruction to increase boot disk size when the host has less free space

- Add `run_randomized_benchmarks.sh` to run each benchmark once per explicit shuffle seed with GOMAXPROCS=1 and -count=1 - Replace fixed-order `-count=10` runs with 20 per-seed runs for reproducible before-and-after comparisons - Require the script for all performance comparisons in AGENTS.md to prevent fixed-order bias - Document the script usage and default benchmark set in scripts/README.md

- Fuse two consecutive unary reductions into one dispatch boundary (REDUCE_CHAIN) when every predecessor edge lands in the same sole-unary-reduce row - Fuse a reduction followed by a shift into one dispatch boundary (REDUCE_SHIFT) when the reduce-then-shift landing row is static across all predecessor edges - Gate both superinstructions behind GTS_C4_REDUCE_CHAIN and GTS_C4_REDUCE_SHIFT environment variables while correctness receipts mature - Add a direct shift fast path that skips ClassifiedBoundary construction for corridor-proven shifts - Route single-header reductions through a trusted-corridor path that skips action-row revalidation and condense-candidate collection - Tighten the sparse-table binary search in corridorDispatch to use bitmask alignment instead of per-iteration multiplication

- Copy `resultErrorSummary` and `resultCompatibilityApplied` from the old tree to the reused tree to prevent loss of finalized result state - Verify preservation of these fields in `TestReuseTreeWithNewSourceKeepsPrimaryArena`

…o avoid redundant lookups - Store the subtree record in materializationPostorderFrame when pushing, so the traversal reads it from the frame instead of calling c.subtree again on pop - Clear the full frame capacity in Reset to release retained subtree record pointers

…ath counters - Inline reduceChildBuildItemForEntry into appendReduceChildrenToScratch to remove a per-child function call from the reduce hot path - Remove unused childCount, parentVisible, and lang parameters from buildReduceChildrenAllVisible and appendReduceChildrenToScratch - Pass symbol visibility as an explicit parameter to appendReduceChildItemToScratch to avoid recomputing it - Add perf counters for reduce child build paths (empty, all-visible, scratch-no-alias, scratch-general) to attribute build cost - Report new build path counters in the warm materialization benchmark

- Add --wall-timeout flag to run_parity_in_docker.sh that stops the container after a wall time limit and returns exit code 124 on expiry - Record wall timeout state, log exit code, and container exit code in metadata output - Add VM max-run-duration, Spot price cap, and timeout environment variables to the v10 GCP runbook - Add checkpoint monitoring procedure to project final duration and stop runs that exceed the wall limit

- Add a 262144-entry temporary memo tier that activates when recovery parses thrash the standard cache, retaining the standard cache for restoration after the parse completes - Share the parser bounded recovery memo with GLR merge cost comparisons and drop the unbounded fallback map when activated - Track parse operation depth so nested operations keep the temporary tier alive until the outermost operation ends - Record peak tier and collision count per tree for bounded recovery-memo observability - Shrink cNodeMemoCacheEntry from 32 to 24 bytes on amd64 by narrowing visCount to uint32 - Move uncommon recovery state into a lazy parserColdState sidecar to preserve the hot Parser layout

… replay - Add a memoization cache for syntheticRootReplayAdvanceToken keyed on (top, lookahead) to skip redundant advance computations during incremental replay - Store cached frame results in fixed-size pages (16K frames per page, 64 pages max) that bound total frame storage to 4 MiB and avoid stream-growth copies - Pack page index, offset, and output count into a single uint32 span to keep each cache entry at 4 bytes - Add 15 perf counters to track cache attempts, hits, misses, stores, cap skips, and output size distribution for profiling - Add TestSyntheticRootReplayAdvanceMemoUsesPackedStream to verify span packing round-trips and cache hits return stored frames without growing the page stream

- Add opt-in Lean 4 grammar authored with the Go DSL. The grammar targets Lean 4.32.2 and provides stable nodes for core declarations. Extension-specific syntax remains in line-scoped custom_command nodes because Lean modules extend the parser at runtime. - Add external scanner for nested block comments, documentation comments, and module documentation comments. - Add highlight captures and outline tags queries for editor integration. - Extend RegisterExtension to accept GrammarSource and TagsQuery fields. An empty GrammarSource defaults to GrammarSourceGrammargen. This lets external packages declare blob-based grammars and symbol extraction without calling Register directly. - Add corpus characterization test that validates 3,699 files from the official Lean 4.32.2 source without recovery nodes or early stops. - Keep the grammar opt-in to preserve the default 206-language graduation gates.

- Add stage M0 merge-event census that counts stack-merge events on both the reference C runtime and the production Go parser to measure the merge gap - Instrument the production merge path in glr.go behind the gts_merge_census build tag so the default build carries zero cost and an inert test ratchets that - Add a merge-event-census-cgo CI lane that builds the instrumented reference runtime from the pinned module-cache source and runs the census in Docker - Share the census corpus definition with the stage-D0 derivation-set differential so both censuses measure the same 104 constructed sources - Pin the M0 baseline (M_p/M_c = 15/191) and add the Foo[int](a) discriminator witness that spec.merge-time-election.v1 section 3 demands - Document the new gate in docs/ci-gate-coverage.md and flag the stage-D0 derivation-set census as a dark gate whose pin has already drifted from 32 to 24

- Regenerate compiled grammar binary for Lean parser

- Replace per-entry slice allocations in the close memo with fixed-size page storage, mirroring the existing advance stream paging pattern - Pack page index, offset, and output count into a 4-byte span value so the memo map stores one word per entry instead of a heap-allocated slice - Bound close frame storage to 16 MiB across 256 pages of 16384 frames each and skip memoization when capacity is full - Store zero-length closures as a zero span to avoid page allocation for empty results - Add performance counters for close output distribution, peak output size, and memo capacity skip events - Extend tests to verify packed span encoding round-trips and memo bookkeeping invariants

- Remove Language struct cloning for retry mode overrides - Restore original retry profile after the benchmark runs - Reduce allocations during benchmark initialization

- Guard replaceBest against a candidate that is already the incumbent tree - Without the guard, re-ranking releases the incumbent and clears the selected tree - Add V runtime profile that exercises the accepted-error retry ladder path

- Update CHANGELOG.md release date to 2026-08-13 and add entries for Lean 4 support, V10 fleet harness, randomized benchmark improvements, merge-event census, scanner fixes, retry corrections, bytecode corridor extensions, and replay memoization gains - Clarify in README.md that v0.49.0 is a candidate pending CI and soak, with v0.48.1 as the latest immutable release - Update bytecode-stream-spec.md to note that the stage 2 corridor compiler and VM are implemented and opt-in via GTS_C4_CORRIDOR=1 - Add BenchmarkKDLRecoveryGarbageSuffix and BenchmarkExpectedRootCanFrameLongRepeat to the randomized benchmark script default set - Update scripts/README.md to document the expanded benchmark set including recovery and replay targets

- Specify explicit event types for pull request triggers - Run workflows only on open, synchronize, reopen, and ready for review events - Prevent unnecessary runs on draft edits or metadata changes

- Compile definition, call, heritage, and import extraction into a 16-bit instruction stream - Reduce inspection time with a single-pass traversal - Add benchmarks and tests to verify parity with legacy extractors - Update documentation and benchmark scripts for the new extraction lanes

odvcencio added 30 commits August 1, 2026 04:19

add(docs): Add GCP fleet runbook and bytecode spec index

db8941a

- Add runbook for v10 full-fleet performance measurement on GCP - Document VM provisioning, corpus authentication, and bisect ladder - Add index file that points to the canonical bytecode stream specification

fix(docs): fix corpus materialization path in GCP runbook

6648559

- Change directory to `cgo_harness` before running the corpus source tool - Update the relative path for `real_corpus_sources` to match the new working directory

update(docs): correct disk space guidance in fleet runbook

467aa7f

- Change free-space requirement from 100 GB before scan to 40 GB after materializing the corpus - Add instruction to increase boot disk size when the host has less free space

fix(parser): bound accepted-error retry escalation

e3a90f0

merge: reconcile origin/main with retry fixes

eacdc6e

improve(v): reuse the certified clean-wide result

610d97d

fix(perf): certify assertion-required C scanners

ef74835

instrument(perf): record peak resident memory

f461568

fix(perf): isolate cgo oracle admission

05d2ca1

fix(perf): retain dual parser stops

112e7d1

perf(parser): iterate single derivation paths

5003ffb

perf(parser): cache external subtree provenance

e72438a

perf(parser): pool external replay token sources

b3fd2c2

perf(parser): reuse synthetic replay scratch

14b257e

perf(parser): reuse replay closure scratch

f9b26a3

perf(parser): hash synthetic replay frames

941cab0

perf(parser): stop doomed compact node growth

1c070c9

odvcencio added 7 commits August 8, 2026 21:29

perf(parser): hash synthetic replay gap cursors

aeb02e7

perf(parser): memoize synthetic replay gap tokens

86171a5

update(lean): Update Lean grammar binary

e47f12b

- Regenerate compiled grammar binary for Lean parser

odvcencio added 3 commits August 8, 2026 23:54

refactor(bench): remove language cloning in recovery benchmark

2983fb2

- Remove Language struct cloning for retry mode overrides - Restore original retry profile after the benchmark runs - Reduce allocations during benchmark initialization

odvcencio added the area/api Public APIs and consumer-facing interfaces label Aug 9, 2026

odvcencio marked this pull request as ready for review August 9, 2026 07:57

odvcencio added 2 commits August 9, 2026 00:58

update(ci): Restrict pull request trigger types

b686215

- Specify explicit event types for pull request triggers - Run workflows only on open, synchronize, reopen, and ready for review events - Prevent unnecessary runs on draft edits or metadata changes

odvcencio mentioned this pull request Aug 9, 2026

Feature request: language-neutral file outline API #649

Open

odvcencio merged commit e1ce651 into main Aug 9, 2026
44 checks passed

Navigation Menu

Release v0.49.0: reconcile V10 parser and performance train - #665

Release v0.49.0: reconcile V10 parser and performance train#665
odvcencio merged 42 commits into
mainfrom
codex/linkerscript-normalizer-retirement

odvcencio commented Aug 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

odvcencio commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Net changeset map

Functional changes

Parser, recovery, and compact execution

Fact extraction API

Grammar, measurement, and release controls

Performance evidence

Accepted V10 epoch

FactProgram

Parser and replay work

Merge-event evidence

Correctness evidence

Issue and pull request linkage

Mined follow-on targets

Release reconciliation

Benchmark method

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

odvcencio commented Aug 9, 2026 •

edited

Loading