Skip to content

fix(standards/cpp): support functor operator() and type conversion operators in func_start regex - #1759

Merged
squid-protocol merged 6 commits into
squid-protocol:mainfrom
emre155:fix/cpp-operator-conversion-support
Aug 17, 2026
Merged

fix(standards/cpp): support functor operator() and type conversion operators in func_start regex#1759
squid-protocol merged 6 commits into
squid-protocol:mainfrom
emre155:fix/cpp-operator-conversion-support

Conversation

@emre155

@emre155 emre155 commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Addresses part of #1752

Summary of Changes

  1. Return-type Shield: Added negative lookahead (?!(?:[a-zA-Z_]\w*::)*operator\b) to the preceding return-type loop in func_start. This prevents the return-type loop from greedily consuming MyClass::operator when encountering non-primitive conversion operators (such as MyClass::operator std::string() or MyClass::operator Foo()).
  2. Identifier Capture: Correctly captures functor operator(), symbol operators, memory operators, and type conversion operators (including primitive, leading-::, and namespace-qualified targets).
  3. Regression Tests: Added test cases to tests/extraction/languages/test_cpp.py covering operator(), operator bool(), operator std::string(), and operator Foo().

Golden Crucible Updates:
The golden master fixtures were regenerated because the new C++ operator regex now correctly identifies conversion operators like Variant::operator ::AABB and Variant::operator String which were previously missed. This caused expected cascading shifts in the topological coordinates and mass metrics for the C++ corpus.

Golden Crucible Updates:
The golden master fixtures were regenerated because the new C++ operator regex now correctly identifies conversion operators like Variant::operator ::AABB and Variant::operator String which were previously missed. This caused expected cascading shifts in the topological coordinates and mass metrics for the C++ corpus.

Copilot AI lite review requested due to automatic review settings August 16, 2026 14:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@squid-protocol squid-protocol left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on, Yunus — operator() and type-conversion operators being invisible to func_start was a real gap, and I appreciate you tracking down the actual root cause in issue #1752 rather than just patching the symptom.

I tested the new alternative against a few cases and wanted to flag one before merging. The three examples from #1752 all work correctly:

  • operator() → captures MyClass::operator()
  • operator bool() → captures MyClass::operator bool
  • operator ::AABB() → captures MyClass::operator ::AABB

But for the "namespace-qualified conversions" case the PR description also calls out, the captured name is wrong:

>>> rx.search("MyClass::operator std::string() const {\n").group(1)
'std::string'          # expected: 'MyClass::operator std::string'

>>> rx.search("MyClass::operator Foo() const {\n").group(1)
'Foo'                   # expected: 'MyClass::operator Foo'

It still matches (so recall goes up, which is the point of the issue), but the earlier "return type" loop in the pattern (a few lines above the identifier capture) is greedy and doesn't recognize operator as special — for any conversion target that isn't one of the hardcoded primitive keywords (bool, int, void, etc.), that loop consumes ClassName::operator as if it were an ordinary return-type token, and the new alternative never gets a chance to fire. What lands in group 1 is just the trailing type name, silently dropped of both the class qualifier and the operator keyword — so it ends up misattributed as an unrelated function/identifier (e.g. std::string) rather than the conversion operator. bool and the leading-:: case both happen to dodge this because they're either in that keyword exclusion list or can't be parsed as a bare qualified identifier, so they force the loop to back off — but std::string, or a plain custom class name like Foo, don't get that protection.

Could you extend the "not a function" shield (or add a similar guard) so the return-type loop doesn't eat the operator token itself? Also, since this touches func_start for a widely-used language, it'd be good to add a regression case to tests/extraction/languages/test_cpp.py covering at least a non-primitive conversion type (operator std::string() or similar) so this doesn't regress silently.

One more small thing: the PR says "Closes #1752", but that issue also lists two other bugs (C++ digit-separator string-shielding, and #else macro-shielding) that this PR doesn't address — might be worth switching to "part of #1752" or similar so the issue doesn't auto-close before those are fixed too.

Happy to help iterate on the regex if useful — this is close, and the recall improvement is a nice find either way. Thanks again for digging into this!

…_start regex

When parsing non-primitive conversion operators like MyClass::operator std::string(), the return type loop previously consumed MyClass::operator
as a return type prefix, leaving only std::string captured as the function
name. Add a negative lookahead to prevent the return type loop from consuming
operator identifiers.

Addresses part of squid-protocol#1752
@emre155

emre155 commented Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

@squid-protocol Thanks for the detailed feedback!

I've updated the regex and pushed the fix:

  1. Added a negative lookahead (?!(?:[a-zA-Z_]\w*::)*operator\b) to the return-type loop (Step 4) so it stops before consuming ClassName::operator.
  2. Verified all conversion operator patterns:
    • MyClass::operator()() const -> MyClass::operator()
    • MyClass::operator bool() const -> MyClass::operator bool
    • MyClass::operator ::AABB() const -> MyClass::operator ::AABB
    • MyClass::operator std::string() const -> MyClass::operator std::string
    • MyClass::operator Foo() const -> MyClass::operator Foo
  3. Added unit tests for these cases in ests/extraction/languages/test_cpp.py.
  4. Updated PR reference to "Addresses part of C++ func_recall_pct: Missing Support for Type Conversion Operators and Functors #1752".

@squid-protocol squid-protocol left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — I tested the negative lookahead against all the cases from before, including the ones that were broken:

>>> rx.search("MyClass::operator std::string() const {\n").group(1)
'MyClass::operator std::string'   # was 'std::string'
>>> rx.search("MyClass::operator Foo() const {\n").group(1)
'MyClass::operator Foo'           # was 'Foo'

All correctly capture the full qualified name now, and the ordinary cases (plain functions, constructors, operator+, operator new) are still unaffected. No ReDoS regression from the new lookahead either (linear up to n=50k). Nice fix.

I also approved the pending workflow runs on this branch — as a first-time contributor's PR, GitHub was holding CI at "action_required" and nothing had actually run yet. It's executing now; I'll merge once it's green.

Thanks again for the thorough follow-through, Yunus!

@squid-protocol

Copy link
Copy Markdown
Owner

Looked at the three failures — they need two different responses, not one.

crucible-audit (both full-precision and zero-dependency): expected, please regenerate. The mismatches are all tiny floating-point shifts in 3D topological coordinates (Expected 7206.07, Got 7206.09) scattered across files that have nothing to do with C++ (JSON, XML, unrelated repos). That's the normal ripple when any file's structural signature count changes anywhere in the corpus — the whole coordinate space renormalizes, every entity inherits a tiny shift. Since this PR is a real func_start recall improvement, this drift is expected. Please run:

python tests/tools/update_golden_master.py

and add a short note to the PR description on why the fixtures changed (see CONTRIBUTING.md's "Updating the Golden Crucible Baseline" section) so it's not invisible in the diff.

tree-sitter-accuracy-audit: please don't just --regenerate this one — it found something real.

tree_sitter_accuracy_audit: 2 regression(s) against the baseline:
  extra_functions: 55 -> 62 (lower is better, this got worse)
  args_exact_match: 1202 -> 1200 (higher is better, this got worse)

(The "MISSING CLASSES IN godot/variant.h" line right above that is unrelated noise, for what it's worth — I checked, those are template <> struct HashMapComparatorDefault<Variant> {...}-style explicit specializations, a pre-existing class_start gap that has nothing to do with operators or this diff. It shows up in other PRs' runs too.)

The two numeric regressions are real and specific to this PR — I diffed the branch against main and this is the only change in the C++ rules. Something in the reordered operator alternatives or the new (?!(?:[a-zA-Z_]\w*::)*operator\b) lookahead is now matching 7 things elsewhere in the corpus that aren't real functions, and losing exact arg-count matches on 2 others. Could you run python tests/tools/tree_sitter_accuracy_audit.py --lang cpp locally and see which specific file/function is now miscounted? Once that's root-caused, regenerating both baselines together should be a single clean pass.

@squid-protocol

squid-protocol commented Aug 17, 2026

Copy link
Copy Markdown
Owner

@emre155 Thank you so much for your hard work on this, and congratulations on your first contribution! I went ahead and pushed the final fixes for the remaining CI issues (the tree-sitter regressions and the downstream ruff/mypy checks) directly to your branch, since the instructions for resolving the tree-sitter accuracy audit regressions weren't very clear initially. I really appreciate your time and effort getting this started! This is now fully green and ready for merge.

@squid-protocol
squid-protocol force-pushed the fix/cpp-operator-conversion-support branch from 7a4784f to c3d8bec Compare August 17, 2026 01:40
@squid-protocol
squid-protocol merged commit 424bdea into squid-protocol:main Aug 17, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants