fix(parsers): remove file-size threshold via iterative walk + keep_alive (closes #163)

[Wolfvin]

Closes #163

Race Condition Notice

I claimed this issue at 2026-07-02T18:16:06Z (54 seconds after it was created). Another worker claimed the same issue at 18:24:28Z (8 minutes AFTER my claim) and submitted PR #168. Per the precedent set by issue #54 (PRs #144/#145), I am submitting my PR as a competing approach for BOS to decide.

Summary

Removes the conservative file-size threshold (100 lines JS / 200 lines Python) that was silently skipping the most complex files in real codebases. The segfault root cause from issue #116 is addressed via iterative DFS walk + explicit Tree reference + Node keep_alive list. Files above 10,000 lines (absolute hard limit) are explicitly skipped with a skipped[] entry — not silently.

Approach Comparison with PR #168

Aspect	PR #168	This PR
Threshold	250 (JS) / 500 (PY)	10,000 (both)
Large files (>threshold)	Regex fallback (partial coverage)	Tree-sitter (full AST)
Root cause fix	No — assumes segfault unfixable	Yes — iterative walk + keep_alive
Works on tree-sitter 0.25.x	Not tested	Yes (verified)
Works on tree-sitter 0.26.0	Yes (via fallback)	No (0.26 has separate regression)
skipped[] field	skipped_from_tree_sitter	skipped (file_too_large reason)
Iterative walk	JS only	JS + Python

Root Cause Analysis

PR #168 concludes: "tree-sitter 0.26 binding has a fundamental SIGSEGV bug on large files that cannot be fully mitigated from Python."

My investigation found this is specific to tree-sitter 0.26.0. With tree-sitter 0.25.2 (the version referenced in issue #116 and consistent with pyproject.toml >=0.21.0), the segfault does NOT occur with:

1. parse_tree() holding the Tree reference explicitly
2. Iterative DFS walk (no Python recursion across function boundaries)
3. keep_alive list pinning every visited Node for the walk duration
4. gc.disable() during parse + walk (already in place)

Verified by stress-testing on tree-sitter 0.25.2:

• Python: 8,504 lines (2,200 nodes, 2,000 edges) ✓
• Python: deeply-nested depth-100 (101 nodes) ✓
• Python: 3,504 lines (500 functions) ✓
• JS: 5,001 lines (1,000 functions with calls) ✓
• JS: 2,001 lines (2,000 simple functions) ✓
• JS: 1,201 lines (200 functions with deep callback nesting) ✓

Changes

• python_parser.py: remove MAX_SAFE_PY_LINES=200; convert recursive _walk to iterative DFS with explicit (node, depth, class_name, fn_id) stack and keep_alive list; use parse_tree() to hold Tree reference; add ABSOLUTE_HARD_LIMIT_LINES=10000 with explicit skipped[] entry; always include skipped[] in return.
• js_backend_parser.py: remove MAX_SAFE_JS_LINES=100; convert recursive _walk AND _walk_calls to iterative DFS with keep_alive; add ABSOLUTE_HARD_LIMIT_LINES=10000 with explicit skipped[] entry; always include skipped[] in return.
• scan.py: aggregate skipped[] entries from all parsers into a top-level skipped_files list in the scan result.
• tests/test_large_file_parsing.py: 11 new tests covering files above old threshold, 3000-line DoD, deeply-nested depth-100, skipped[] field presence, and explicit skip at hard limit.

Definition of Done Status

• Root cause of segfault on large files confirmed (tree-sitter 0.26 regression; 0.25.x works with proper Tree pinning)
• _last_tree fix applied uniformly (already in base_parser.parse and parse_tree; python_parser now uses parse_tree directly)
• Verified: files up to 3,000 lines parse without SIGSEGV (tested up to 8,504 lines Python, 5,001 lines JS)
• File size threshold raised to 10,000 (≥ 5,000 per DoD)
• Silent skip replaced with explicit skipped[] list in JSON output
• [~] complexity on Wolfvin/Regrets detects regret.js:main — cannot test without the repo, but threshold removal makes it visible (full AST, not regex fallback)
• No regression on existing tests (1067 passed, 76 skipped, 0 failed; 1 pre-existing failure in test_codelensignore unrelated)

Findings

1. tree-sitter 0.26.0 regression: The segfault that PR fix(parsers): replace silent skip with regex fallback for large files (closes #163) #168 attributes to "fundamental binding bug" is specific to 0.26.0. Version 0.25.2 does not exhibit this bug with proper Tree pinning. BOS may want to consider pinning tree-sitter<0.26 in pyproject.toml until the 0.26 regression is fixed upstream.

2. Iterative walk is necessary regardless: Even on 0.25.x, deeply-nested files (depth ≥ 100) can segfault with recursive walk. The iterative DFS + keep_alive pattern from base_parser.walk_tree should be the standard for all tree-sitter parsers.

3. PR fix(parsers): replace silent skip with regex fallback for large files (closes #163) #168 tests validate.js / validate.py — note these are different files from the issue spec (regret.js). The issue specifically mentions scripts/regret.js from Wolfvin/Regrets (2,731 lines, cyclomatic 338).

Test Results

tests/test_large_file_parsing.py: 11 passed
Full suite (excluding pre-existing failure): 1067 passed, 76 skipped, 0 failed

Sebelum: [python_parser.py / js_backend_parser.py silently returned {nodes:[], edges:[]} for files > 100/200 lines]
Sesudah: [parsers return full AST for files up to 10,000 lines; files above get explicit skipped[] entry]

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sonarqubecloud]

Quality Gate passed

Issues
10 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud