fix(prism): stop treating C++ digit separators as char-literal openers

[uuzzrm]

What

C++14+ lets you write a single quote inside numeric literals as a digit separator: 512'000, 1'000'000'000, 0xDE'AD'BE'EF. GitGalaxy's literal shields didn't know about that. Their single-quote branches are unbounded, so a separator quote was read as the start of a char literal and paired with the next stray quote anywhere later in the file — swallowing real // and /* */ comments (prism) and blanking real function bodies (detector brace scan) in between.

The fix (two layers)

1. prism.py — CPP_LITERAL_MASK_PATTERN for the generic standard_block stripper: consume digit separators as their own alternative ([0-9a-fA-F]'[0-9a-fA-F]) so they can never open a span, and bound the char-literal branch to 10 chars (same bound prism.py: _strip_nested_comments' unbounded single-quote shield swallows huge code spans (scala/rust/swift/haskell/scheme) #1302 already applies to the recursive_block shield). Kept per-language because JS/PHP single-quoted strings legitimately span many characters and keep the unbounded shared pattern.

2. detector.py — _build_brace_safe_stream's own single-quote shield had the identical bug: a separator quote blanked every real function body between it and the next apostrophe, which showed up as a tree-sitter accuracy regression (cpp found_functions 1376 → 1350) once prism stopped swallowing the comment that used to terminate the span. Same bounded pattern applied here, mirroring the existing rust/zig bound.

Verification

• 7 new regression tests (test_prism_issue_1718.py, test_detector_issue_1718.py): far-away quote pairing, hex/multi-group separators, a comment apostrophe near a separator, real char literals still shielding, u8 prefixed literals, JS/PHP strings staying whole, and detector brace-scan survival.
• Full suite passes locally; golden crucible passes in both full-precision and zero-dependency modes.
• Tree-sitter accuracy audit: no regressions for cpp (the detector fix restores the 26 functions that were temporarily lost).

Why the golden masters changed

Regenerated both fixtures via tests/tools/update_golden_master.py. The diff is confined to cpp/godot/editor_node.cpp and its derived aggregates: Coding LOC 7678 → 7665, Documentation LOC 441 → 454 (the // 500 KB comment after 512'000 is now correctly stripped instead of being swallowed into a fake literal), plus the corresponding shifts in structural signatures and exposures. No other file in the corpus changed.

Closes #1718.

[uuzzrm]

Closes #1718.

[squid-protocol]

Thanks for chasing this down, Ruiming — the digit-separator root cause and the editor_node.cpp before/after are a clean diagnosis, and I like that the fix stays scoped to C++ instead of touching the shared SHIELD_PATTERN that JS/PHP rely on.

CI is currently red on two checks that look like real substance rather than flakiness, though, so I don't think this is mergeable yet:

tree-sitter-accuracy-audit fails with a measured C++ regression:

=== cpp ===
MISSING CLASSES IN godot/variant.h: {'HashMapComparatorDefault<Variant>', 'is_zero_constructible<Variant>'}
tree_sitter_accuracy_audit: 2 regression(s) against the baseline:
  found_functions: 1376 -> 1350 (higher is better, this got worse)
  args_exact_match: 1202 -> 1186 (higher is better, this got worse)

godot/variant.h is a different file from the one this PR targets, and it's now losing two template-specialization classes it used to detect correctly, plus 26 fewer functions found across the C++ corpus overall. That's at odds with the PR description's claim that "No other file in the corpus changed."

crucible-audit (zero-dependency) also fails, while crucible-audit (full-precision) passes — mismatches show up in unrelated files across many languages (python/airflow, dart/flutter, php/laravel, zig, swift...), which reads as global rank/blast-radius ripple rather than direct C++ damage, but it's still a real CI gate failure. Combined with the PR saying "Golden crucible passes in both full-precision and zero-dependency modes" locally, this looks like it could be the stale-venv footgun CLAUDE.md's Differential Scan section calls out (PR #579/#723): a crucible_check.py venv whose editable install silently resolves to a different checkout gives a false "zero diff" pass locally while CI correctly fails. Worth confirming with python -c "import gitgalaxy; print(gitgalaxy.__file__)" inside whatever venv ran the local check.

For the func/args regression itself: I haven't nailed the exact trigger, but two things in CPP_LITERAL_MASK_PATTERN look like plausible candidates worth checking against variant.h directly:

• [0-9a-fA-F]'[0-9a-fA-F] isn't anchored to an actual numeric-literal context — it's just three characters in that shape, so it could coincidentally fire near an ordinary hex-valued char literal ('a', 'f', etc.) sitting next to another hex-ish character, not just real 512'000-style separators.
• The {0,10} bound on the char-literal branch — C++'s universal-character-name escapes (\uXXXX, \UXXXXXXXX, and C++23's \N{...}) can run longer than 10 characters between the quotes, so a real (if rare) literal could get clipped and desync the shield the same way the original unbounded version did, just in the other direction.

Could you re-run tree_sitter_accuracy_audit.py --all and crucible_check.py (fresh venv, not a reused one) and see what's actually happening in variant.h? Happy to look again once those are green.

[uuzzrm]

Chased this down - the prism shield was only half the fix. The detector has its own single-quote literal pass (_build_brace_safe_stream) that still treated a separator quote as a char-literal opener and paired it with the next stray apostrophe anywhere later in the file. Reproduced the exact CI regression locally against the pinned v1.0 corpus:

found_functions: 1376 -> 1350
args_exact_match: 1202 -> 1186

with variant.h losing HashMapComparatorDefault and is_zero_constructible.

Applied the same bounded pattern to the detector's shield (digit separator consumed as its own alternative, char-literal branch bounded to 10 chars, same shape as the prism mask). Re-ran the tree-sitter audit against the corpus: no regressions. Added a regression test (separator followed by functions, comment apostrophe, real char literal), reblessed both golden masters via update_golden_master.py, and the golden crucible test passes against them. Full detector suite is green.

Tree-sitter audit and crucible should be green on CI now.

[uuzzrm]

Thanks for the thorough review — and for the specific pointers on where to look. I went through each point; here's what I found.

The two red checks were against the prism-only state, before the detector fix landed. Your review is pinned to commit 6fd7a0ee; the follow-up commits (91911020 detector shield + 8bbfc761 golden master rebless) went up right after. On the current head 1c173f29, all checks pass — tree-sitter-accuracy-audit, crucible-audit in both modes, full-suite, ruff/mypy/dead-key. I re-ran the cpp audit locally too and got "no regressions" against the committed baseline.

On variant.h and the "26 fewer functions" — I believe that was the intermediate prism-only regression, now resolved, and the two variant.h classes are pre-existing misses. I checked the committed baseline first: it records found_classes: 138 vs real_classes: 140, so exactly those two classes were already not detected before this PR. I then ran the audit against main itself (pre-change) and it prints the identical MISSING CLASSES IN godot/variant.h: {'HashMapComparatorDefault<Variant>', 'is_zero_constructible<Variant>'} line. The current head measures found_functions 1376 -> 1380 and args_exact_match 1202 -> 1206 — the fix actually improves recall rather than regressing it, because the detector shield stops the separator quote from blanking real function bodies.

Your two pattern concerns, addressed directly:

1. [0-9a-fA-F]'[0-9a-fA-F] not anchored to numeric-literal context. I agree it's a three-char shape rather than a full numeric-literal parse, so I tested it against the realistic contexts where it could false-fire: hex char literals ('a', 'f'), hex escapes ('\x41'), u8'x' / L'a' prefixes, char literals next to hex literals/indices/case labels. None of them produce a match — the pattern only fires on actual separators (512'000, 0xDE'AD'BE'EF), because in valid C++ a ' immediately flanked by two hex digits is a digit separator, not a char literal. I've added a comment noting the reasoning. If you'd still rather see it tightened to a lookbehind/lookahead form I can do that.

2. {0,10} bound clipping long escapes. This one is legitimate — I'd copied the prism.py: _strip_nested_comments' unbounded single-quote shield swallows huge code spans (scala/rust/swift/haskell/scheme) #1302 bound without thinking about C++23 named character escapes (\N{LATIN CAPITAL LETTER A}, which can run 20+ chars and contains braces). The corpus doesn't currently exercise them, but a real literal would get clipped and desync the shield the same way the original unbounded version did, just in the other direction. I've widened the C++-only bound to {0,64} in both prism.py and detector.py — comfortably covers every named escape, still far too short for any cross-file cascade — and added a regression test with a \N{...} literal plus a trailing comment. The recursive_block shield keeps its own 10-char bound, and JS/PHP keep the unbounded shared pattern.

On the stale-venv footgun: you're right that the earlier local "zero diff" was misleading. The zero-dependency fixture in the first push had been regenerated with a subprocess galaxyscope that resolved to the wrong env. I redid both fixtures with the venv's Scripts dir on PATH and re-ran the crucible in both modes — they pass against the current fixtures (and the committed zero-dep fixture now correctly reports Zero-Dependency Mode Active: true).

The bound-hardening is in 1c173f29; CI is re-running now. Happy to make further changes if anything still doesn't look right.

[squid-protocol]

CI's fully green now and the fix itself checks out — thanks for the thorough follow-up on variant.h, the stale venv, and the false-positive testing. Not blocking on these, but two small things worth a quick pass before merge:

1. Stale comment in detector.py. The last commit widened the bound from {0,10} to {0,64} and updated the comment above CPP_LITERAL_MASK_PATTERN in prism.py to match, but the parallel comment in detector.py still says "bound the branch to 10 chars" right above code that now uses {0,64}:

   if lang_id in ("cpp", "c"):
       # #1718: ... Consume separators as their own
       # alternative (same shape as prism.py's CPP_LITERAL_MASK_PATTERN) and bound the branch
       # to 10 chars, matching #1302/#1426.
       single_quote = r"[0-9a-fA-F]'[0-9a-fA-F]|(?<!\\)'(?:\\.|[^'\\]){0,64}'"

   test_detector_issue_1718.py's module docstring has the same "10 chars" line.

2. Test coverage is asymmetric. test_prism_issue_1718.py got a new regression test for the C++23 named-escape case (test_issue_1718_cpp23_named_escape_literal_stays_intact), but test_detector_issue_1718.py didn't get the equivalent test even though detector.py's shield was widened for the identical reason. Given the earlier fix was exactly about detector.py having its own independent copy of this logic that needed its own verification, it'd be good for it to have its own test for this case too rather than relying on prism's test to vouch for both.

Small optional one: detector.py's fix applies to lang_id in ("cpp", "c"), but prism.py's only applies to "cpp". Probably harmless either way (C doesn't have digit separators, and C char literals are short too), but worth a one-line confirmation that the C inclusion in detector.py is intentional rather than a copy-paste artifact from the cpp/c grouping used elsewhere in that function.

[squid-protocol]

LGTM! I pushed the fixes and resolved the conflicts. Approving.