feat(find_symbol): report searched kinds, index module/class variables#68
Closed
andrebrait wants to merge 353 commits into
Closed
feat(find_symbol): report searched kinds, index module/class variables#68andrebrait wants to merge 353 commits into
andrebrait wants to merge 353 commits into
Conversation
…ty pre-warming - PPM (Prediction by Partial Matching) up to order-5 with escape-probability blending over the first-order fallback. Persisted alongside the existing Markov table and wired into beam_search_continuations for richer speculation. - Token Economy ROI: compute_observation_roi uses exponential decay of P(hit) times per-type multiplier; run_roi_gc archives negative-ROI observations (dry_run by default). memory_roi_gc + memory_roi_stats tools, monthly GC via SessionStart hook. - Leiden community pre-warming: greedy modularity maximization on the symbol dependency graph runs after reindex. get_function_source emits a 🏘️ community footer and _warm_cache_async prefetches community peers when the cluster is small. get_community + get_leiden_stats tools. Tool count: 86 → 90. 891/891 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… abstractions Rissanen MDL: replace N similar observations with 1 abstraction + N deltas when L(abstraction) + Σ L(delta) < Σ L(obs). - mdl_distiller.py: Jaccard agglomerative clustering per type, compact abstraction ([MDL:type] core_tokens — representative), delta_encode strips shared tokens so each member stores only its novel fragments. - memory_db.run_mdl_distillation: dry_run preview or apply — creates type=convention decay_immune abstraction, rewrites members to "[delta] ...\n[abstraction_id: N]", tags 'mdl-distilled', inserts 'supersedes' observation_links. - memory_distill tool + ts memory distill CLI. - SessionEnd hook suggests distillation when >20 undistilled obs have candidates. - get_usage_stats: MDL line (abstractions + distilled count). Tool count: 90 → 91. 891/891 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PPM variable-order Markov (order 2-5) Token Economy ROI GC Leiden community pre-warming MDL memory distillation LinUCB contextual bandit injection (10-feature model) Cross-session warm start (cosine signature matching) Self-consistency check (Bayesian validity, quarantine) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrates @joshorig's Java support branch (feat/java-language-support) on top of v2.3.0. Resolves Mibayy#10. Adds: - tree-sitter-based java_annotator (classes, interfaces, enums, records, constructors, methods, annotations, qualified names) - gradle_annotator for build files - Java dependency graph (invocations, constructor edges, method refs, lambda paths, interface→implementation, Spring-aware entry edges) - Java coverage in find_symbol / get_dependencies / get_dependents / get_change_impact / get_call_chain / get_routes / get_entry_points / find_dead_code / find_impacted_test_files - 3 new MCP tools: find_allocation_hotspots, find_performance_hotspots, get_duplicate_classes (tool total: 95 → 98) Rebased conflicts resolved: - tests/test_tool_schemas.py count 89 → 98 Dependencies added (hard): - tree-sitter >=0.25,<0.26 - tree-sitter-java >=0.23,<0.24 Tests: 1006/1006 pass (891 existing + 115 new Java tests). Ruff clean on all new files. Follow-ups tracked separately (per PR Mibayy#11 author's own notes): - find_performance_hotspots too broad on .get() patterns - get_class_source(level=2) semantic summary quality Co-Authored-By: joshorig <joshorig@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extracted run_migrations() with per-path idempotency (called at MCP startup + lazily from get_db as safety net for tests). - get_db() is now minimal: just opens a WAL connection, no schema work. - Added @contextmanager db_session() guaranteeing close() on exception. - Refactored WRITE functions to use db_session(): observation_save, session_start, session_end, run_decay, run_roi_gc, run_consistency_check, run_mdl_distillation, auto_link_observation, update_consistency_score. - Migrations now run after executescript (fixes fresh-DB ordering bug: ALTER TABLE sessions ADD COLUMN end_type was skipped when sessions table didn't yet exist). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add _META_HANDLERS dispatch table with 18 _hm_* handlers (meta tools). - Extract _track_call, _maybe_compress, _prefetch_next helpers. - call_tool: 395 → 38 lines; flat handler dispatch (meta/memory/slot/qfn). - 1006/1006 tests pass, ruff clean.
…on, utils) - New db_core.py: MEMORY_DB_PATH, run_migrations, get_db, db_session, _now_iso, _now_epoch, _json_dumps, observation_hash, strip_private, relative_age, _fts5_safe_query. 201 lines, zero higher-level deps. - memory_db.py re-exports the public surface + thin wrappers for get_db / db_session / run_migrations that read memory_db.MEMORY_DB_PATH at call time, so test patching continues to work. - 1006/1006 tests pass, ruff clean. Step B partial: obs_store / session_store / memory_ops extraction deferred — those require unwinding tight cross-function coupling across ~80 functions and 7 importers; acceptable follow-up given current win already isolates the SQLite plumbing.
Adds 33 tests covering v2.x feature surface that lacked direct coverage (LinUCB, SessionWarmStart, Consistency, MDL, PPM, TCA, Leiden). Tool manifest compressed from 51620 to 41531 chars (-19.5%) by trimming verbose descriptions and parameter hints on the top verbose tool schemas. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- TOKEN_SAVIOR_STATS_DIR env var overrides ~/.local/share/token-savior in server.py, slot_manager.py, query_api.py (dashboard.py already did) - tests/conftest.py sets TOKEN_SAVIOR_STATS_DIR to a fresh tempdir at import time, before any test module loads server/slot_manager — kills the 196 orphan test_*.json leak cleanly - type hints: get_db/db_session/run_migrations wrappers in memory_db.py now return sqlite3.Connection / AbstractContextManager[Connection] - bulk dict/list generic fixes: bare `dict` → `dict[str, Any]`, bare `list` → `list[Any]` across memory_db.py + server.py mypy errors: memory_db 170 → 41 (-76%), server 162 → 122 (-25%) tests: 1039/1039 passing, ruff clean, 0 prod state pollution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add pip install with [mcp] extra to Quick Start - Add claude mcp add one-liner - Fix binary name in YAML example (token-savior-recall -> token-savior) - Clarify .mcp.json vs settings.json scoping Closes Mibayy#12 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-step refactor sweep — 1069/1069 tests green throughout. Shared utilities: - output_helpers.py: extracted _build_line_offsets + _truncate_output (used by impacted_tests.py, project_actions.py) - brace_matcher.py: rewrote find_brace_end_csharp with explicit while-loop state; the original `for idx in range(...)` rebound idx every iteration, losing the verbatim-string skip and reading } as code (covered by characterization tests in tests/test_brace_matcher.py) server.py — _format_usage_stats: - 278-line function with depth 6 → 33-line orchestrator + 18 _usage_* section helpers (each ≤50 lines, depth ≤3) query_api.py — create_file_query_functions: - 247-line factory with 13 nested closures → 26-line dispatcher using functools.partial + 13 module-level _file_*_impl helpers - Public dict shape and callable signatures unchanged Annotator changes (java/gradle): adopt the new output_helpers utility.
Move all module-level mutable globals from server.py to a new src/token_savior/server_state.py. Pure relocation, zero behavior change. server.py now imports state via: from token_savior import server_state as s from token_savior.server_state import server All references to moved globals (counters, caches, engine instances, slot manager, tool-set frozensets) are routed via `s.<name>`. Local variables that previously shadowed `s` are renamed (`stats`, `summary`, `sess`, `sug`) to keep the module alias unambiguous. Tests updated to mutate scalar state through server_state directly, since rebinding via the server alias would no longer propagate. 1069 tests pass.
Move cross-cutting request-cycle helpers out of server.py: - _fmt_lines, compress_symbol_output (compact-output formatting) - _detect_client_name, _parse_workspace_roots, _register_roots - _get_stats_file, _load_cumulative_stats, _flush_stats - _format_result, _count_and_wrap_result, _estimate_naive_chars_for_call - _prep, _warm_cache_async, _recompute_leiden, _resolve_project_root - _TOOL_COST_MULTIPLIERS, _CLIENT_NAME, _SESSION_LABEL constants server.py re-exports them so existing call sites and tests stay working. Drops imports that were only feeding the moved code (Server, SlotManager, PPMPrefetcher, TCAEngine, LeidenCommunities, LinUCBInjector, SessionWarmStart, Path, hashlib, threading, uuid). server.py: 3232 → 2732 lines. 1069 tests pass.
First handler split: move _h_discover_project_actions and _h_run_project_action to server_handlers/project_actions.py, exported via a HANDLERS dict that gets spread into the existing _SLOT_HANDLERS table. Adds the server_handlers package skeleton (__init__.py) that step 13 will use to aggregate ALL_HANDLERS with collision detection. 1069 tests pass.
Move the four git-flavored handlers (get_git_status, get_changed_symbols, summarize_patch_by_symbol, build_commit_summary) to server_handlers/git.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused git_tracker / git_ops / compact_ops imports from server.py. 1069 tests pass.
Move four edit-flavored handlers (replace_symbol_source, insert_near_symbol, verify_edit, apply_symbol_change_and_validate) into server_handlers/edit.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused edit_ops / workflow_ops imports from server.py. 1069 tests pass.
Move the six checkpoint handlers (create/list/delete/prune/restore/ compare) into server_handlers/checkpoints.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused checkpoint_ops import from server.py. 1069 tests pass.
Move the two test-impact handlers (find_impacted_test_files, run_impacted_tests) into server_handlers/tests.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused impacted_tests import from server.py. 1069 tests pass.
Move the eight code-quality and analysis handlers (analyze_config, find_dead_code, find_hotspots, find_allocation_hotspots, find_performance_hotspots, detect_breaking_changes, find_cross_project_deps, analyze_docker) into server_handlers/analysis.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused breaking_changes/complexity/config_analyzer/ cross_project/dead_code/docker_analyzer/java_quality/ProjectIndex imports from server.py. Incidentally fixes a latent shadowing bug in _h_find_cross_project_deps where the loop variable s shadowed the module alias s; the new module renames the loop variable to sibling_slot, mirroring _h_find_dead_code. 1069 tests pass.
Move the four project-lifecycle manager handlers (list_projects, switch_project, set_project_root, reindex) into server_handlers/project.py with a HANDLERS dict spread into _META_HANDLERS. The new module references state._slot_mgr explicitly instead of relying on the server.py module alias `s`, matching the pattern of the other extracted handler modules. 1069 tests pass.
Move the 28-entry _QFN_HANDLERS dispatch table plus the CSC (Compact Symbol Cache) subsystem (_lookup_symbol_meta, _csc_compact_response, _csc_diff_preview, _csc_maybe_serve) and the three named query handlers (_q_get_class_source, _q_get_function_source, _q_get_edit_context) into server_handlers/code_nav.py. server.py imports the dict as QFN_HANDLERS and re-exports _q_get_edit_context for tests/test_server.py. Drops the redundant s._PREFETCHABLE_TOOLS reassignment from server.py since server_state.py already defines the same frozenset. 1069 tests pass.
Move the 18 _usage_* section builders, _format_usage_stats, _format_duration, and the 12 _hm_get_* meta-handlers (get_usage_stats, get_session_budget, get_coactive_symbols, get_tca_stats, get_dcp_stats, get_community, get_linucb_stats, get_warmstart_stats, get_leiden_stats, get_speculation_stats, get_lattice_stats, get_call_predictions) into server_handlers/stats.py. server.py shrinks from 1860 to 1314 lines. The stats handlers are spread into _META_HANDLERS via **_STATS_HANDLERS. _format_duration and _format_usage_stats are re-exported from server.py so tests/test_usage_stats.py keeps working. _hm_get_session_budget uses a lazy import for _resolve_memory_project (still in server.py until step 12). 1069 tests pass.
Move 26 _mh_* memory tool handlers, 2 _hm_* admin handlers (memory_consistency, memory_quarantine_list), and 4 helpers (_resolve_memory_project, _invalidate_injection_hash, _linucb_credit_reward, _build_linucb_context) into server_handlers/memory.py. server.py shrinks from 1314 to 260 lines. stats.py now imports _resolve_memory_project eagerly from memory.py (no more lazy import workaround). _resolve_memory_project is re-exported from server.py for backward compatibility. 1069 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add server_handlers/__init__.py that merges per-domain HANDLERS dicts into four call-shape categories (META, MEMORY, SLOT, QFN) and asserts the union is disjoint, so a tool name accidentally added to two domains fails at import time rather than silently shadowing. server.py now imports the four aggregated dispatch tables from the package, eliminating the inline _META_HANDLERS / _SLOT_HANDLERS dict constructions. Re-exports of _q_get_edit_context, _resolve_memory_project, _format_duration, and _format_usage_stats kept for tests. 1069 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Drop unused imports (json, os, time) and unused server_runtime re-exports (_CLIENT_NAME, _SESSION_LABEL, _TOOL_COST_MULTIPLIERS, _detect_client_name, _estimate_naive_chars_for_call, _fmt_lines, _get_stats_file, _recompute_leiden, _resolve_project_root) that no caller imports anymore. - Drop _resolve_memory_project re-export (only used internally). - Restore missing 'import sys' that was accidentally removed during the server_runtime extraction; the exception handler in call_tool references sys.stderr but no caller had hit that path in tests so the broken import sat dormant. - Sort imports and drop the now-stale section-comment headers. server.py: 230 -> 202 lines. 1069 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ffset bug
python_annotator carried its own _compute_line_offsets helper that called
source.splitlines(keepends=False) and then incremented offsets by len(line)+1.
On CRLF input that arithmetic is off by one per line: splitlines() strips the
trailing \r so the +1 only accounts for the \n, leaving every line offset 1
byte short cumulatively.
Drop the private helper, import build_line_char_offsets from models, and split
with source.split('\n') -- the convention used by ~16 other annotators. That
preserves the trailing \r inside each line so len(line)+1 correctly accounts
for the full \r\n pair.
Verified on CRLF input 'x = 1\r\ny = 2\r\nz = 3\r\n' (21 chars):
before: offsets = [0, 6, 12] (y at 6, actual position 7 -- wrong)
after: offsets = [0, 7, 14, 21] (matches str.index lookups exactly)
Two test_markup_python edge-case tests asserted the legacy splitlines() line
count for empty source (0) and trailing-newline source (2). With split('\n')
those become 1 and 3 respectively, which now matches every other annotator's
total_lines convention. Tests updated with explanatory comments.
1069 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the 90-line monolithic `find_brace_end_rust` into a 36-line driver
plus four focused helpers, mirroring the csharp refactor pattern. Also
fixes a latent `for idx in range` rebind bug where multi-line raw-string
advancement was lost when control returned to the outer loop (the new
driver uses a single `while idx < len(lines)` loop).
Helpers extracted:
- _rust_skip_block_comment — handles nested /* ... */
- _rust_try_skip_raw_string — r#..."..."#... with EOF handling
- _rust_skip_string — regular "..." with backslash escapes
- _rust_skip_char_or_lifetime — 'a' / '\n' vs lifetime 'a
LOC: 90 → 36 (driver) + 4 helpers (~85 LOC total, including docstrings)
Depth: ~5 → ~3 (driver), ≤3 in helpers
Tests: +26 characterization cases (TestFindBraceEndRust), all green;
full brace_matcher suite 56/56 passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…eholder v4.3.0 ts init was broken on vanilla pip install: hooks/ was missing from the wheel and JSON configs had hard-coded paths to the dev machine. v4.3.1 fixes both via hatch force-include + placeholder substitution in _load_hook_bundles().
v3.5.0..v4.3.2: every successful tool call returned isError=True with a CallToolResult pydantic validation error. The shim TextContent from _compat.py reached the SDK without being converted to the real mcp.types.TextContent (same class name, different class object, pydantic v2 rejects it). list_tools already converted ToolDef -> mcp.types.Tool at the protocol boundary. Mirror that pattern for TextContent in call_tool via _to_mcp_content(). Cold-start cost preserved -- the mcp.types import stays lazy and server-only. Test gap closed: prior call_tool integration tests inspected the returned list directly, never going through SDK validation. New test builds a real CallToolResult, catching shim leaks on success path, error path, and meta-tool paths (ts_search, ts_extended). Reported by @zinkovsky in Mibayy#32.
9-day usage audit (2026-05-17..26, 869 tool calls) surfaced 3 wasteful patterns. CLAUDE.md and trailing _hints were not enough -- the agent ignored them. Move the corrections to where they're hard to miss. 1) Chain nudges (call_tool) 42 find_symbol(X) -> get_function_source(X) and 26 find_symbol(X) -> get_full_context(X) chains on the same symbol within 60s, all foldable into one get_full_context call. Detector: rolling 8-call buffer per server, 60s window. On match, prepend a [NUDGE] TextContent block at the top of the response so it survives downstream compression. Opt-out: TOKEN_SAVIOR_CHAIN_NUDGE=0. 2) ts_search warm-up (warm_up_async) ts_search averaged 4867ms over 19 calls; the Nomic load + 66 tool description embeddings dominate the first call. Spawn a background thread at server startup so the first client call hits a populated cache. Cold load stays silent on failure (substring fallback exists). Opt-out: TOKEN_SAVIOR_NO_WARMUP=1. 3) set_project_root nudge The cheap path already exists (32% of calls hit it). Tighten the response so the agent learns to call switch_project next time -- that's the documented entry point in CLAUDE.md and one round-trip lighter than set_project_root. Tests: 1764 passed (12 new). test_chain_nudge fixture cleans up tmp_path slots from _slot_mgr to avoid polluting memory_viewer's active-project resolution.
…/9d) 9-day data showed 187 occurrences of get_function_source(X) or get_class_source(X) followed by get_full_context(X) within 60s. The first read is wasted -- get_full_context re-fetches the source as part of its bundle. Extends the detector built in v4.4.0 (42 cases of find_symbol -> get_function_source) with a second pattern, multiplying the practical coverage by ~5x. Also snapshot _tool_call_counts in the chain-nudge test fixture so the suite doesn't push the global past the 15-call navigation-overuse threshold and flip code_nav._stop_hint behavior in test_query_api.
- persist registered projects + cache-aware set/switch_project (kills set_project_root churn: 51 calls, 14.6s outlier, collector-crypt-scanner 20x) - nudge get_edit_context before edits (0/199 adoption) - nudge ts_execute on 5+ nav calls in a window (41 calls despite thousands unitary) - persist ts_search tool-desc embeddings (cold start 5.7s -> skips re-embed) - silence no-observations hook spam (3578 lines in hook-errors.log) 1771 tests pass (+6 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- daemon_client.py: minimal Unix-socket client (best-effort, None on failure) - _handle_ts_search delegates the first cold call to a warm ts _daemon-serve (~130ms) instead of the ~5.7s in-process Nomic load; opt-in via TS_SEARCH_COLD_DELEGATE=1, falls back to in-process on any daemon failure - cli._daemon_serve routes ts_search through its handler (was 'unknown tool') 1779 tests pass (+8 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…4.0) The hardcoded string sat at 3.4.0 through the entire v4.x line, which masked that the daily-driver venv was running a stale build. Now sourced from importlib.metadata so it tracks pyproject.toml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mon delegation - scripts/ts_audit.py: automates the usage audit (latency, chains, adoption gaps, nudge fires, ML liveness); baseline snapshot for measuring nudge effect - telemetry.record_nudge/nudge_counts: persist per-kind nudge fires so effectiveness can be tracked vs adoption over successive audits - TCA revival: flush_session was never called (session_count stuck at 0 all deployment) -> now flushed at switch_project + atexit. Broken feature, not dead code; made live instead of removed - ts-daemon.service: launch venv python (has fastembed) so delegated ts_search is embedding-quality warm (~23ms) instead of substring 1786 tests pass (+7 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- server_handlers/resources.py: list + read observations as MCP resources (bounded, ranked by memory_index score) - wired in main() via list_resources/read_resource, opt-out TS_RESOURCES_DISABLED=1 - read-only, additive; tool path untouched 1792 tests pass (+6 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit over 425 real sessions: get_edit_context 0/219 edits, its nudge fired 12x and converted 0. Stop nudging; fold the value into the edit result. _edit_impact_notice() appends callers + impacted tests to every successful symbol edit (reuses get_dependents + find_impacted_test_files). Pattern-3 nudge retired. Opt out with TOKEN_SAVIOR_EDIT_IMPACT=0. 1790 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
find_symbol only ever searched functions and classes, but a miss returned `complete: True` and its docstring told callers they "do not need to fall back to grep/search_codebase". For anything that is not a def or a class — a module global, a constant, a class attribute — that is a confidently wrong answer. `pfb`, a module-level dict referenced 317 times in one file, reported as "symbol not found, scanned_files: 2085, complete: true". Two changes: 1. A miss now reports `searched` (the kinds actually scanned) and, when an indexed kind was left out, `not_searched` plus a ready-to-run `retry_with` call. `complete: True` stays on the hit path, where it is true. The MCP `_suggestion` now names the retry before sending the caller to grep. 2. `find_symbol(name, kinds=[...])` accepts `function`, `class`, and `variable`; variables cover module globals, constants (UPPER_SNAKE or `Final[...]`), and class attributes. The Python annotator extracts them from `Assign`/`AnnAssign` at module and class scope — function locals are skipped, since they are not addressable from another file. `AugAssign` is skipped too: it rebinds rather than defines. Bindings live in a separate `ProjectIndex.variable_table` (name and qualified name -> defining files) rather than `symbol_table`, so a global can never shadow a same-named function in the resolvers that walk the symbol table, and same-named globals across files report as ambiguous instead of silently picking one. `TOKEN_SAVIOR_VARIABLES` gates both halves: `off` skips extraction and rejects an explicit `kinds=["variable"]` (rather than returning an empty result that reads as "no such variable"), `index` (default) extracts but searches only on request, `search` also puts variables in the default kinds. The new `variables` field is persisted, so `_CACHE_VERSION` goes 2 -> 3; a stale v2 cache would otherwise report every project as variable-free. Other annotators leave `variables` empty for now, which `searched` / `not_searched` reports honestly rather than hiding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
andrebrait
added a commit
to pfBlockerNG/pfBlockerNG
that referenced
this pull request
Jul 22, 2026
Picks up two fork commits. `find_symbol` now reports which symbol kinds it actually searched instead of claiming an exhaustive scan on a miss, and it can search module-level and class-level variables via `find_symbol(name, kinds=["variable"])` — previously only functions and classes were ever indexed, so a global like `$pfb` / `pfb` returned "not found, complete: true". Python, PHP, and shell annotators all extract variables. Indexed by default; searched on request. `TOKEN_SAVIOR_VARIABLES=search` puts them in the default kinds, `off` disables both halves. Indexing this repo yields 4255 variables across 861 files. The first session after this lands rebuilds the shared venv, as any TS_SOURCE change does. Upstream: Mibayy/token-savior#68 (Python half; the PHP and shell halves depend on the not-yet-merged annotator PRs #50/#51). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
|
Merged as 30daf13 on Not via the merge button, and the reason matters: Full suite green (2044 passed) and CI green on If you send more PRs, please rebase on the current Thanks for the work, it was accurate and well tested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
find_symbolsearches functions and classes — and only those._resolve_symbol_inforeadsmeta.functionsandmeta.classeson both its symbol-table fast path and its all-files fallback, andStructuralMetadatahas no field for anything else. That has been true since the first commit; there is no variables field to consult.The failure mode is not the missing coverage, it is what a miss claims:
{"error": "symbol 'pfb' not found", "scanned_files": 2085, "complete": true}plus a docstring promising callers they "do not need to fall back to grep/search_codebase".
pfbin that project is a module-leveldictreferenced 317 times in a single file. An agent that trustscomplete: trueconcludes the symbol does not exist and stops looking.Changes
1. Honest miss reporting. A miss now returns the kinds it actually searched, and — when an indexed kind was excluded — what it skipped plus the exact retry:
{ "error": "symbol 'pfb' not found", "scanned_files": 2085, "searched": ["function", "class"], "not_searched": ["variable"], "retry_with": "find_symbol(name='pfb', kinds=['variable'])" }complete: truestays on the hit path, where it is accurate. The MCP_suggestionleads with the retry instead of sending the caller straight to grep.2.
kindsparameter and variable indexing.find_symbol(name, kinds=[...])takesfunction,class, andvariable. Variables cover module globals, constants (UPPER_SNAKE or aFinal[...]annotation), and class attributes. The Python annotator extracts them fromAssign/AnnAssignat module and class scope. Function locals are skipped — they cannot be referenced from another module, so indexing them would grow the index without making anything findable.AugAssignis skipped as well:x += 1rebinds an existing name, so treating it as a definition would report the wrong line.Bindings go in a new
ProjectIndex.variable_table(bare name and qualified name → defining files), deliberately not insymbol_table:symbol_table;logger,DEFAULTS) are routine, so the value is a list and a collision reports as ambiguous rather than silently picking a winner.3.
TOKEN_SAVIOR_VARIABLESgates both halves:offkinds=["variable"]is rejected with an explanation, rather than returning an empty result that reads as "no such variable"index(default)searchRead per call, so it can be flipped without a server restart.
Compatibility
_CACHE_VERSION2 → 3. The newvariablesfield is persisted; a stale v2 cache would report every project as variable-free.kindsis the previous behaviour, so existing callers see no change apart from the richer miss payload.kindsonly when the caller sets it, so query engines with the older two-argumentfind_symbolsignature keep working.variablesempty.searched/not_searchedreport that honestly instead of hiding it. Adding extraction to another annotator is now a per-language change with no further plumbing.Tests
tests/test_symbol_kinds.py, 24 tests: miss reporting, extraction (module/class scope, constant detection, locals excluded),kindslookup and isolation, ambiguity, all three env-var modes, cache round-trip, and a guard that the cache version was bumped. Full suite: 1813 passed, 1 pre-existing failure (test_daemon_client.py::test_successful_call_returns_text— pytest'stmp_pathexceeds the 104-byte macOSAF_UNIXlimit; unrelated to this change and green on the same machine under a shorter path).ruff check src/ tests/clean.One doc gap left deliberately: the new variable is documented in the
variables_mode()docstring rather than the README, since the README has no environment-variable reference yet (#56).