fix(parsers): replace silent skip with regex fallback for large files (closes #163)

[Wolfvin]

Closes #163

Problem

CodeLens silently skipped files above hardcoded line thresholds to avoid a tree-sitter 0.26 binding SIGSEGV (tracked in #116):

• JavaScript: files > 100 lines → skipped (returned {"nodes": [], "edges": []})
• Python: files > 200 lines → skipped

This caused the most complex files in a codebase — the ones most worth analyzing — to be invisible to all downstream engines (complexity, dead-code, smell, entrypoints). On Wolfvin/Regrets ~40% of the codebase was silently dropped, including the worst hotspots:

• scripts/validate.js (2730 lines) → silently returned 0 nodes
• scripts/validate.py (2361 lines) → silently returned 0 nodes

Root Cause Investigation

The tree-sitter 0.26 Python binding has a nondeterministic SIGSEGV on large files. Existing mitigations (BaseParser._last_tree, parse_tree(), _gc.disable()) reduce crash frequency but do not eliminate it. Verified by stress-testing:

• Synthetic JS files: 250 lines parse OK, 275+ lines SIGSEGV (nondeterministic, ~50% crash rate at 300 lines)
• Synthetic Python files: 500 lines parse OK, 1000+ lines SIGSEGV (nondeterministic)
• Real-world scripts/validate.js (2730 lines) → SIGSEGV on every attempt
• Real-world scripts/validate.py (2361 lines) → SIGSEGV on every attempt

The bug cannot be fixed from Python — it requires a binding upgrade (tracked in #116).

Fix

Instead of silently skipping, large files now use the REGEX FALLBACK parser (parse_js_backend_fallback / parse_python_fallback), which gives partial coverage (function declarations + direct calls) instead of zero coverage. The result includes a skipped_from_tree_sitter field so callers know tree-sitter was not used and why.

The scan command aggregates all such entries into a top-level skipped_from_tree_sitter list in its JSON output.

Additional hardening to JSBackendParser

• Iterative DFS walk replaces recursive _walk (prevents Python stack frames from holding stale Node references across function boundaries — reduces crash frequency on smaller files).
• Iterative DFS in _find_calls_in_scope for the same reason.

outline_engine.py

No longer preemptively falls back to regex for JS > 100 lines or Python > 200 lines — attempts tree-sitter first, falls back only on actual parse exception.

Thresholds raised

• JS: 100 → 250 (largest value that passed 5 consecutive stress-test runs without SIGSEGV)
• Python: 200 → 500 (same methodology)

Definition of Done Status

• Root cause of segfault on large files confirmed (tree-sitter 0.26 binding bug, not GC issue — GC mitigations are already in place)
• _last_tree fix applied uniformly across all parser classes (verified — already in base_parser.py)
• Verified: files up to 3,000 lines parse without SIGSEGV — CANNOT be met due to tree-sitter 0.26 binding bug. Files above ~250 lines (JS) / ~500 lines (PY) use regex fallback instead. Full AST parsing requires binding upgrade ([RED TEAM] scan command segfaults on CodeLens repo itself — recursive walk_tree crashes (JS & Python parsers) #116).
• File size threshold raised (100→250 for JS, 200→500 for PY — could not reach ≥5000 due to binding bug)
• Silent skip replaced with explicit skipped_from_tree_sitter list in JSON output
• complexity on Wolfvin/Regrets now detects validate.js and validate.py (via regex fallback — partial coverage instead of zero)
• No regression on existing tests (1380 passed, 76 skipped, 1 pre-existing failure in test_codelensignore::TestBackwardCompat::test_actual_target_dir_is_ignored unrelated to this change)

Files Changed

• scripts/parsers/js_backend_parser.py — threshold + regex fallback + iterative walk
• scripts/parsers/python_parser.py — threshold + regex fallback
• scripts/outline_engine.py — remove preemptive threshold; fall back only on actual exception
• scripts/commands/scan.py — aggregate skipped_from_tree_sitter into top-level output field
• tests/test_large_file_fallback.py — new tests verifying fallback behavior
• CHANGELOG.md — entry documenting the fix

Tests

tests/test_large_file_fallback.py: 6 passed
Full suite: 1380 passed, 76 skipped, 1 failed (pre-existing, unrelated)

Findings

The tree-sitter 0.26 Python binding has a fundamental SIGSEGV bug on large files that cannot be fully mitigated from Python. The original MAX_SAFE_JS_LINES = 100 / MAX_SAFE_PY_LINES = 200 thresholds were conservative but necessary. This PR raises them as far as safely possible and replaces silent skip with explicit fallback + reporting. A permanent fix requires upgrading the tree-sitter binding (issue #116).

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Wolfvin]

Closing — #169 is the superior fix. It removes the threshold entirely via iterative walk + keep_alive (root-cause fix), vs this PR which only raises thresholds (band-aid). #169 has been merged.