fix(callgraph): extract module-top-level calls to fix rc undercount (closes #210)

[Wolfvin]

Closes #210

Patch Peta (delta struktural)

Files:

• scripts/parsers/ts_backend_parser.py — add _find_module_level_calls method; call it from extract_references after the per-function pass to extract module-top-level calls and calls inside inline arrow callbacks (issue fix(callgraph): reference_count severely undercounts cross-file calls #210 root-cause fix for TS backend).
• scripts/parsers/js_backend_parser.py — add _find_module_level_calls method (iterative DFS + keep_alive pattern per issue [RED TEAM] scan command segfaults on CodeLens repo itself — recursive walk_tree crashes (JS & Python parsers) #116/[BUG] File size threshold too conservative — large files silently skipped, breaking analysis completeness #163); call it from extract_references after the main walk loop (issue fix(callgraph): reference_count severely undercounts cross-file calls #210 root-cause fix for JS backend).
• scripts/parsers/fallback_js_backend.py — add module-level scan for lines not covered by any function's approximate scope; emit edges with synthetic <module> source (issue fix(callgraph): reference_count severely undercounts cross-file calls #210 fix for regex fallback path).
• tests/test_js_backend_parser.py — add TestIssue210ModuleLevelCalls class with 6 regression tests covering middleware-factory argument, inline arrow callback body, multi-site accumulation, no-double-count, sanity.
• tests/test_ts_backend_parser.py — new file, 10 tests: baseline TS parsing + issue fix(callgraph): reference_count severely undercounts cross-file calls #210 regression coverage (TS is the primary fix target since the KDS backend is TypeScript).
• docs/design/0210-module-level-call-extraction.md — design doc documenting root cause, fix approach, and 3 alternatives considered + rejected.

Endpoints: none
Schema: none (graph_edges table gets more rows but no schema change; source_id values now include <file>:0:<module> synthetic ids mirroring the existing <rel_path>:0:file convention from hybrid_type_resolver._file_node_id)
Konvensi: Module-top-level calls now produce CALLS edges with synthetic source_id = "<file>:0:<module>" and NO corresponding graph_nodes entry. This is the same dangling-source convention already used for IMPORTS edges. Devs running SQLite queries on graph_edges should be aware that source_id values ending in :0:<module> or :0:file are synthetic and don't JOIN to graph_nodes.

Klaim + Bukti

Root cause identified and documented. Issue #210 suspected callgraph_engine.py (_parse_calls_treesitter ~line 1589, _resolve_call ~line 2055) but explicitly instructed independent investigation. Actual root cause is in scripts/parsers/ts_backend_parser.py and scripts/parsers/js_backend_parser.py (NOT callgraph_engine.py — that file is only used by dataflow command, not by scan→search --mode symbol pipeline). The per-function pass in both parsers only walks bodies of registered function/class declarations. Two patterns slip through:

1. Module-top-level calls (e.g. router.post(path, requirePermission('admin'), handler) at file root) — never extracted because no function body contains them.
2. Calls inside inline arrow/function-expression callbacks (e.g. (req, res) => { hasPermission(...) } passed as argument) — the arrow function isn't registered as a function declaration, so its body is never walked.

Reproduction confirmed BEFORE fix (via /home/z/my-project/work/scripts/diag_ts_parser.py running TSBackendParser directly on a 6-file fixture mirroring the KDS backend patterns):

requirePermission ref_count = [0] (expected 12)
hasPermission       ref_count = [1] (expected 13)

Reproduction confirmed AFTER fix (same diagnostic script, same fixture):

requirePermission ref_count = [12] (expected 12)
hasPermission       ref_count = [13] (expected 13)

DoD point 1 — search "requirePermission" <repro-path> --mode symbol rc >= 13 (or actual call site count):

$ /home/z/my-project/work/venv/bin/python scripts/codelens.py search "requirePermission" /home/z/my-project/work/repro --mode symbol --format json | grep ref_count
      "ref_count": 12,

The repro fixture has 12 real call sites (3 per route file × 4 route files), and rc=12 matches exactly. For the KDS backend reported in the issue (13+ call sites), rc would be 13+.

DoD point 2 — search "hasPermission" <repro-path> --mode symbol rc == 7 (or actual call site count):

$ /home/z/my-project/work/venv/bin/python scripts/codelens.py search "hasPermission" /home/z/my-project/work/repro --mode symbol --format json | grep ref_count
      "ref_count": 13,

The repro fixture has 13 real call sites (3 per route file × 4 route files + 1 inside requirePermission's returned arrow function body), and rc=13 matches exactly. For the KDS backend reported in the issue (7 call sites), rc would be 7.

SQLite verification (issue's verification method, corrected): The issue used SELECT * FROM graph_edges WHERE target_id LIKE '%requirePermission%' which returns 0 rows because target_id uses file:line format (e.g. src/middleware/permission-gate.ts:7), not function names. Correct query:

$ /home/z/my-project/work/venv/bin/python -c "import sqlite3; c=sqlite3.connect('/home/z/my-project/work/repro/.codelens/codelens.db'); print('requirePermission CALLS edges:', c.execute(\"SELECT COUNT(*) FROM graph_edges WHERE edge_type='CALLS' AND target_id='src/middleware/permission-gate.ts:7'\").fetchone()[0]); print('hasPermission CALLS edges:', c.execute(\"SELECT COUNT(*) FROM graph_edges WHERE edge_type='CALLS' AND target_id='src/lib/permissions.ts:6'\").fetchone()[0])"
requirePermission CALLS edges: 12
hasPermission CALLS edges: 13

Sample edge:

{'source_id': 'src/routes/task-templates.ts:0:<module>', 'target_id': 'src/middleware/permission-gate.ts:7', 'file': 'src/routes/task-templates.ts', 'line': 0}

DoD point 4 — pytest tests/ -k "callgraph or ref_count" 0 new regressions:

$ /home/z/my-project/work/venv/bin/python -m pytest tests/ -k "callgraph or ref_count" --ignore=tests/test_integration.py ... 
1 passed, 2248 deselected in 2.31s

DoD point 5 — new unit tests reproducing both cases:

$ /home/z/my-project/work/venv/bin/python -m pytest tests/test_js_backend_parser.py tests/test_ts_backend_parser.py -k "Issue210" --tb=no
12 passed, 19 deselected in 0.08s

Tests cover: middleware-factory argument pattern, inline arrow callback body pattern, multi-site accumulation (no dedup collapse), no-double-count for calls inside registered function bodies, no-double-count for class method bodies, no-double-count for exported arrow functions, sanity extraction of router.post itself.

Full test suite — no new regressions:

$ /home/z/my-project/work/venv/bin/python -m pytest tests/ --ignore=tests/test_integration.py --ignore=tests/test_lsp_server.py --ignore=tests/test_rule_engine.py --ignore=tests/test_rule_matcher.py --ignore=tests/test_rule_pattern_parser.py --ignore=tests/test_rule_test.py --ignore=tests/test_rule_validate.py --ignore=tests/test_semantic_engine.py --ignore=tests/test_large_file_parsing.py --tb=no
17 failed, 2136 passed, 96 skipped in 38.96s

Same 17 pre-existing failures as main (test_doctor, test_confidence, test_formatters_phase2, test_node_types — all unrelated to callgraph). 2136 passed = 2120 (main baseline) + 16 new tests added by this PR.

Constraint: dead-code / impact / trace run without crash, numbers more accurate:

$ /home/z/my-project/work/venv/bin/python scripts/codelens.py audit /home/z/my-project/work/repro --check dead-code
"s": "ok", "total_dead_code": 0, "recommended_action": "No dead code detected"

$ /home/z/my-project/work/venv/bin/python scripts/codelens.py impact /home/z/my-project/work/repro --name requirePermission
"s": "ok" (no crash, completes successfully)

$ /home/z/my-project/work/venv/bin/python scripts/codelens.py context /home/z/my-project/work/repro --check trace --name requirePermission --direction up
"s": "ok" (no crash, completes successfully)

All three commands run without crash. dead-code now correctly reports 0 dead code (requirePermission and hasPermission have ref_count > 0 and aren't falsely flagged). impact and trace run without crash; their caller counts for module-level callers remain 0 because they JOIN on source_id = node_id and the synthetic <module> source_id has no matching node — this is NOT a regression (they returned 0 callers before the fix too, because no CALLS edges existed at all). See "Found, not fixed" for follow-up.

Pre-flight checks passed: git fetch origin && git rebase origin/main (up to date, HEAD = 56833c0). 0 open PRs at time of claim (no collision). Recent merged PRs (#207-#213) all touch security/circular/impact/search — none touch callgraph_engine.py or parsers.

Catatan Pendekatan

Issue suspected callgraph_engine.py (_parse_calls_treesitter ~line 1589, _resolve_call ~line 2055) but explicitly instructed independent investigation. Investigation revealed callgraph_engine.py is NOT in the scan → search --mode symbol pipeline — it's only used by dataflow. The actual scan pipeline uses parsers/ts_backend_parser.py (for .ts files) and parsers/js_backend_parser.py (for .js files), with parsers/fallback_js_backend.py as regex fallback. Fix is scoped to these three parser files. Constraint "Jangan ubah command files (commands/*.py) kecuali memang diperlukan" respected — no command files touched.

Synthetic <file>:0:<module> source_id (no corresponding graph_nodes entry) chosen over creating a synthetic <module> node per file, to avoid polluting list/search --mode symbol output with fake function entries. This mirrors the existing convention in hybrid_type_resolver._file_node_id for IMPORTS edges. Trade-off: trace/impact SQLite JOINs on source_id = node_id won't match these synthetic sources, so they don't see module-level callers (same behavior as IMPORTS edges). This is a known limitation, not a regression — see "Found, not fixed" for follow-up.

Breaking / Found-not-fixed

Found, not fixed (Tier 2):

1. trace and impact commands don't see module-level callers of a symbol, because they JOIN graph_edges.source_id against graph_nodes.node_id and the synthetic <file>:0:<module> source_id has no matching node. Before this PR, these commands also returned 0 callers for requirePermission (because no CALLS edges existed at all) — so this is NOT a regression, just a known limitation that persists. Follow-up would require either (a) creating synthetic <module> graph_nodes entries (violates the issue's "no command files" constraint by requiring list.py/search_engine.py filter changes), or (b) updating trace/impact to follow dangling source_ids. Out of scope for issue fix(callgraph): reference_count severely undercounts cross-file calls #210 — BOS to decide if a separate follow-up issue is wanted.

2. tests/test_large_file_parsing.py::test_file_above_old_threshold_is_parsed segfaults at python_parser.py:301 (tree-sitter Python binding issue). PRE-EXISTING on main (verified — fails identically without my changes). Not caused by this PR. Should be added to CONTEXT.md's "Pre-existing failures — selalu ignore" list in a follow-up worker-skills PR.

3. TS backend parser's _find_function_declarations only registers classes via the export_statement branch — non-exported class_declaration nodes are NOT registered. PRE-EXISTING (verified by my test development — had to use export class in the double-count test). Out of scope for issue fix(callgraph): reference_count severely undercounts cross-file calls #210. Could be filed as a separate bug.

4. 17 pre-existing test failures on main (test_doctor, test_confidence, test_formatters_phase2, test_node_types) — all unrelated to callgraph, all present before this PR. Listed for awareness; not caused or worsened by this PR.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.6% Duplication on New Code

See analysis details on SonarQube Cloud