feat(shell): regex-based shell annotator#58
Open
andrebrait wants to merge 352 commits into
Open
Conversation
…ty pre-warming - PPM (Prediction by Partial Matching) up to order-5 with escape-probability blending over the first-order fallback. Persisted alongside the existing Markov table and wired into beam_search_continuations for richer speculation. - Token Economy ROI: compute_observation_roi uses exponential decay of P(hit) times per-type multiplier; run_roi_gc archives negative-ROI observations (dry_run by default). memory_roi_gc + memory_roi_stats tools, monthly GC via SessionStart hook. - Leiden community pre-warming: greedy modularity maximization on the symbol dependency graph runs after reindex. get_function_source emits a 🏘️ community footer and _warm_cache_async prefetches community peers when the cluster is small. get_community + get_leiden_stats tools. Tool count: 86 → 90. 891/891 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… abstractions Rissanen MDL: replace N similar observations with 1 abstraction + N deltas when L(abstraction) + Σ L(delta) < Σ L(obs). - mdl_distiller.py: Jaccard agglomerative clustering per type, compact abstraction ([MDL:type] core_tokens — representative), delta_encode strips shared tokens so each member stores only its novel fragments. - memory_db.run_mdl_distillation: dry_run preview or apply — creates type=convention decay_immune abstraction, rewrites members to "[delta] ...\n[abstraction_id: N]", tags 'mdl-distilled', inserts 'supersedes' observation_links. - memory_distill tool + ts memory distill CLI. - SessionEnd hook suggests distillation when >20 undistilled obs have candidates. - get_usage_stats: MDL line (abstractions + distilled count). Tool count: 90 → 91. 891/891 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PPM variable-order Markov (order 2-5) Token Economy ROI GC Leiden community pre-warming MDL memory distillation LinUCB contextual bandit injection (10-feature model) Cross-session warm start (cosine signature matching) Self-consistency check (Bayesian validity, quarantine) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrates @joshorig's Java support branch (feat/java-language-support) on top of v2.3.0. Resolves Mibayy#10. Adds: - tree-sitter-based java_annotator (classes, interfaces, enums, records, constructors, methods, annotations, qualified names) - gradle_annotator for build files - Java dependency graph (invocations, constructor edges, method refs, lambda paths, interface→implementation, Spring-aware entry edges) - Java coverage in find_symbol / get_dependencies / get_dependents / get_change_impact / get_call_chain / get_routes / get_entry_points / find_dead_code / find_impacted_test_files - 3 new MCP tools: find_allocation_hotspots, find_performance_hotspots, get_duplicate_classes (tool total: 95 → 98) Rebased conflicts resolved: - tests/test_tool_schemas.py count 89 → 98 Dependencies added (hard): - tree-sitter >=0.25,<0.26 - tree-sitter-java >=0.23,<0.24 Tests: 1006/1006 pass (891 existing + 115 new Java tests). Ruff clean on all new files. Follow-ups tracked separately (per PR Mibayy#11 author's own notes): - find_performance_hotspots too broad on .get() patterns - get_class_source(level=2) semantic summary quality Co-Authored-By: joshorig <joshorig@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extracted run_migrations() with per-path idempotency (called at MCP startup + lazily from get_db as safety net for tests). - get_db() is now minimal: just opens a WAL connection, no schema work. - Added @contextmanager db_session() guaranteeing close() on exception. - Refactored WRITE functions to use db_session(): observation_save, session_start, session_end, run_decay, run_roi_gc, run_consistency_check, run_mdl_distillation, auto_link_observation, update_consistency_score. - Migrations now run after executescript (fixes fresh-DB ordering bug: ALTER TABLE sessions ADD COLUMN end_type was skipped when sessions table didn't yet exist). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add _META_HANDLERS dispatch table with 18 _hm_* handlers (meta tools). - Extract _track_call, _maybe_compress, _prefetch_next helpers. - call_tool: 395 → 38 lines; flat handler dispatch (meta/memory/slot/qfn). - 1006/1006 tests pass, ruff clean.
…on, utils) - New db_core.py: MEMORY_DB_PATH, run_migrations, get_db, db_session, _now_iso, _now_epoch, _json_dumps, observation_hash, strip_private, relative_age, _fts5_safe_query. 201 lines, zero higher-level deps. - memory_db.py re-exports the public surface + thin wrappers for get_db / db_session / run_migrations that read memory_db.MEMORY_DB_PATH at call time, so test patching continues to work. - 1006/1006 tests pass, ruff clean. Step B partial: obs_store / session_store / memory_ops extraction deferred — those require unwinding tight cross-function coupling across ~80 functions and 7 importers; acceptable follow-up given current win already isolates the SQLite plumbing.
Adds 33 tests covering v2.x feature surface that lacked direct coverage (LinUCB, SessionWarmStart, Consistency, MDL, PPM, TCA, Leiden). Tool manifest compressed from 51620 to 41531 chars (-19.5%) by trimming verbose descriptions and parameter hints on the top verbose tool schemas. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- TOKEN_SAVIOR_STATS_DIR env var overrides ~/.local/share/token-savior in server.py, slot_manager.py, query_api.py (dashboard.py already did) - tests/conftest.py sets TOKEN_SAVIOR_STATS_DIR to a fresh tempdir at import time, before any test module loads server/slot_manager — kills the 196 orphan test_*.json leak cleanly - type hints: get_db/db_session/run_migrations wrappers in memory_db.py now return sqlite3.Connection / AbstractContextManager[Connection] - bulk dict/list generic fixes: bare `dict` → `dict[str, Any]`, bare `list` → `list[Any]` across memory_db.py + server.py mypy errors: memory_db 170 → 41 (-76%), server 162 → 122 (-25%) tests: 1039/1039 passing, ruff clean, 0 prod state pollution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add pip install with [mcp] extra to Quick Start - Add claude mcp add one-liner - Fix binary name in YAML example (token-savior-recall -> token-savior) - Clarify .mcp.json vs settings.json scoping Closes Mibayy#12 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-step refactor sweep — 1069/1069 tests green throughout. Shared utilities: - output_helpers.py: extracted _build_line_offsets + _truncate_output (used by impacted_tests.py, project_actions.py) - brace_matcher.py: rewrote find_brace_end_csharp with explicit while-loop state; the original `for idx in range(...)` rebound idx every iteration, losing the verbatim-string skip and reading } as code (covered by characterization tests in tests/test_brace_matcher.py) server.py — _format_usage_stats: - 278-line function with depth 6 → 33-line orchestrator + 18 _usage_* section helpers (each ≤50 lines, depth ≤3) query_api.py — create_file_query_functions: - 247-line factory with 13 nested closures → 26-line dispatcher using functools.partial + 13 module-level _file_*_impl helpers - Public dict shape and callable signatures unchanged Annotator changes (java/gradle): adopt the new output_helpers utility.
Move all module-level mutable globals from server.py to a new src/token_savior/server_state.py. Pure relocation, zero behavior change. server.py now imports state via: from token_savior import server_state as s from token_savior.server_state import server All references to moved globals (counters, caches, engine instances, slot manager, tool-set frozensets) are routed via `s.<name>`. Local variables that previously shadowed `s` are renamed (`stats`, `summary`, `sess`, `sug`) to keep the module alias unambiguous. Tests updated to mutate scalar state through server_state directly, since rebinding via the server alias would no longer propagate. 1069 tests pass.
Move cross-cutting request-cycle helpers out of server.py: - _fmt_lines, compress_symbol_output (compact-output formatting) - _detect_client_name, _parse_workspace_roots, _register_roots - _get_stats_file, _load_cumulative_stats, _flush_stats - _format_result, _count_and_wrap_result, _estimate_naive_chars_for_call - _prep, _warm_cache_async, _recompute_leiden, _resolve_project_root - _TOOL_COST_MULTIPLIERS, _CLIENT_NAME, _SESSION_LABEL constants server.py re-exports them so existing call sites and tests stay working. Drops imports that were only feeding the moved code (Server, SlotManager, PPMPrefetcher, TCAEngine, LeidenCommunities, LinUCBInjector, SessionWarmStart, Path, hashlib, threading, uuid). server.py: 3232 → 2732 lines. 1069 tests pass.
First handler split: move _h_discover_project_actions and _h_run_project_action to server_handlers/project_actions.py, exported via a HANDLERS dict that gets spread into the existing _SLOT_HANDLERS table. Adds the server_handlers package skeleton (__init__.py) that step 13 will use to aggregate ALL_HANDLERS with collision detection. 1069 tests pass.
Move the four git-flavored handlers (get_git_status, get_changed_symbols, summarize_patch_by_symbol, build_commit_summary) to server_handlers/git.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused git_tracker / git_ops / compact_ops imports from server.py. 1069 tests pass.
Move four edit-flavored handlers (replace_symbol_source, insert_near_symbol, verify_edit, apply_symbol_change_and_validate) into server_handlers/edit.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused edit_ops / workflow_ops imports from server.py. 1069 tests pass.
Move the six checkpoint handlers (create/list/delete/prune/restore/ compare) into server_handlers/checkpoints.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused checkpoint_ops import from server.py. 1069 tests pass.
Move the two test-impact handlers (find_impacted_test_files, run_impacted_tests) into server_handlers/tests.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused impacted_tests import from server.py. 1069 tests pass.
Move the eight code-quality and analysis handlers (analyze_config, find_dead_code, find_hotspots, find_allocation_hotspots, find_performance_hotspots, detect_breaking_changes, find_cross_project_deps, analyze_docker) into server_handlers/analysis.py with a HANDLERS dict spread into _SLOT_HANDLERS. Drops the now-unused breaking_changes/complexity/config_analyzer/ cross_project/dead_code/docker_analyzer/java_quality/ProjectIndex imports from server.py. Incidentally fixes a latent shadowing bug in _h_find_cross_project_deps where the loop variable s shadowed the module alias s; the new module renames the loop variable to sibling_slot, mirroring _h_find_dead_code. 1069 tests pass.
Move the four project-lifecycle manager handlers (list_projects, switch_project, set_project_root, reindex) into server_handlers/project.py with a HANDLERS dict spread into _META_HANDLERS. The new module references state._slot_mgr explicitly instead of relying on the server.py module alias `s`, matching the pattern of the other extracted handler modules. 1069 tests pass.
Move the 28-entry _QFN_HANDLERS dispatch table plus the CSC (Compact Symbol Cache) subsystem (_lookup_symbol_meta, _csc_compact_response, _csc_diff_preview, _csc_maybe_serve) and the three named query handlers (_q_get_class_source, _q_get_function_source, _q_get_edit_context) into server_handlers/code_nav.py. server.py imports the dict as QFN_HANDLERS and re-exports _q_get_edit_context for tests/test_server.py. Drops the redundant s._PREFETCHABLE_TOOLS reassignment from server.py since server_state.py already defines the same frozenset. 1069 tests pass.
Move the 18 _usage_* section builders, _format_usage_stats, _format_duration, and the 12 _hm_get_* meta-handlers (get_usage_stats, get_session_budget, get_coactive_symbols, get_tca_stats, get_dcp_stats, get_community, get_linucb_stats, get_warmstart_stats, get_leiden_stats, get_speculation_stats, get_lattice_stats, get_call_predictions) into server_handlers/stats.py. server.py shrinks from 1860 to 1314 lines. The stats handlers are spread into _META_HANDLERS via **_STATS_HANDLERS. _format_duration and _format_usage_stats are re-exported from server.py so tests/test_usage_stats.py keeps working. _hm_get_session_budget uses a lazy import for _resolve_memory_project (still in server.py until step 12). 1069 tests pass.
Move 26 _mh_* memory tool handlers, 2 _hm_* admin handlers (memory_consistency, memory_quarantine_list), and 4 helpers (_resolve_memory_project, _invalidate_injection_hash, _linucb_credit_reward, _build_linucb_context) into server_handlers/memory.py. server.py shrinks from 1314 to 260 lines. stats.py now imports _resolve_memory_project eagerly from memory.py (no more lazy import workaround). _resolve_memory_project is re-exported from server.py for backward compatibility. 1069 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add server_handlers/__init__.py that merges per-domain HANDLERS dicts into four call-shape categories (META, MEMORY, SLOT, QFN) and asserts the union is disjoint, so a tool name accidentally added to two domains fails at import time rather than silently shadowing. server.py now imports the four aggregated dispatch tables from the package, eliminating the inline _META_HANDLERS / _SLOT_HANDLERS dict constructions. Re-exports of _q_get_edit_context, _resolve_memory_project, _format_duration, and _format_usage_stats kept for tests. 1069 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Drop unused imports (json, os, time) and unused server_runtime re-exports (_CLIENT_NAME, _SESSION_LABEL, _TOOL_COST_MULTIPLIERS, _detect_client_name, _estimate_naive_chars_for_call, _fmt_lines, _get_stats_file, _recompute_leiden, _resolve_project_root) that no caller imports anymore. - Drop _resolve_memory_project re-export (only used internally). - Restore missing 'import sys' that was accidentally removed during the server_runtime extraction; the exception handler in call_tool references sys.stderr but no caller had hit that path in tests so the broken import sat dormant. - Sort imports and drop the now-stale section-comment headers. server.py: 230 -> 202 lines. 1069 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ffset bug
python_annotator carried its own _compute_line_offsets helper that called
source.splitlines(keepends=False) and then incremented offsets by len(line)+1.
On CRLF input that arithmetic is off by one per line: splitlines() strips the
trailing \r so the +1 only accounts for the \n, leaving every line offset 1
byte short cumulatively.
Drop the private helper, import build_line_char_offsets from models, and split
with source.split('\n') -- the convention used by ~16 other annotators. That
preserves the trailing \r inside each line so len(line)+1 correctly accounts
for the full \r\n pair.
Verified on CRLF input 'x = 1\r\ny = 2\r\nz = 3\r\n' (21 chars):
before: offsets = [0, 6, 12] (y at 6, actual position 7 -- wrong)
after: offsets = [0, 7, 14, 21] (matches str.index lookups exactly)
Two test_markup_python edge-case tests asserted the legacy splitlines() line
count for empty source (0) and trailing-newline source (2). With split('\n')
those become 1 and 3 respectively, which now matches every other annotator's
total_lines convention. Tests updated with explanatory comments.
1069 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…eholder v4.3.0 ts init was broken on vanilla pip install: hooks/ was missing from the wheel and JSON configs had hard-coded paths to the dev machine. v4.3.1 fixes both via hatch force-include + placeholder substitution in _load_hook_bundles().
v3.5.0..v4.3.2: every successful tool call returned isError=True with a CallToolResult pydantic validation error. The shim TextContent from _compat.py reached the SDK without being converted to the real mcp.types.TextContent (same class name, different class object, pydantic v2 rejects it). list_tools already converted ToolDef -> mcp.types.Tool at the protocol boundary. Mirror that pattern for TextContent in call_tool via _to_mcp_content(). Cold-start cost preserved -- the mcp.types import stays lazy and server-only. Test gap closed: prior call_tool integration tests inspected the returned list directly, never going through SDK validation. New test builds a real CallToolResult, catching shim leaks on success path, error path, and meta-tool paths (ts_search, ts_extended). Reported by @zinkovsky in Mibayy#32.
9-day usage audit (2026-05-17..26, 869 tool calls) surfaced 3 wasteful patterns. CLAUDE.md and trailing _hints were not enough -- the agent ignored them. Move the corrections to where they're hard to miss. 1) Chain nudges (call_tool) 42 find_symbol(X) -> get_function_source(X) and 26 find_symbol(X) -> get_full_context(X) chains on the same symbol within 60s, all foldable into one get_full_context call. Detector: rolling 8-call buffer per server, 60s window. On match, prepend a [NUDGE] TextContent block at the top of the response so it survives downstream compression. Opt-out: TOKEN_SAVIOR_CHAIN_NUDGE=0. 2) ts_search warm-up (warm_up_async) ts_search averaged 4867ms over 19 calls; the Nomic load + 66 tool description embeddings dominate the first call. Spawn a background thread at server startup so the first client call hits a populated cache. Cold load stays silent on failure (substring fallback exists). Opt-out: TOKEN_SAVIOR_NO_WARMUP=1. 3) set_project_root nudge The cheap path already exists (32% of calls hit it). Tighten the response so the agent learns to call switch_project next time -- that's the documented entry point in CLAUDE.md and one round-trip lighter than set_project_root. Tests: 1764 passed (12 new). test_chain_nudge fixture cleans up tmp_path slots from _slot_mgr to avoid polluting memory_viewer's active-project resolution.
…/9d) 9-day data showed 187 occurrences of get_function_source(X) or get_class_source(X) followed by get_full_context(X) within 60s. The first read is wasted -- get_full_context re-fetches the source as part of its bundle. Extends the detector built in v4.4.0 (42 cases of find_symbol -> get_function_source) with a second pattern, multiplying the practical coverage by ~5x. Also snapshot _tool_call_counts in the chain-nudge test fixture so the suite doesn't push the global past the 15-call navigation-overuse threshold and flip code_nav._stop_hint behavior in test_query_api.
- persist registered projects + cache-aware set/switch_project (kills set_project_root churn: 51 calls, 14.6s outlier, collector-crypt-scanner 20x) - nudge get_edit_context before edits (0/199 adoption) - nudge ts_execute on 5+ nav calls in a window (41 calls despite thousands unitary) - persist ts_search tool-desc embeddings (cold start 5.7s -> skips re-embed) - silence no-observations hook spam (3578 lines in hook-errors.log) 1771 tests pass (+6 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- daemon_client.py: minimal Unix-socket client (best-effort, None on failure) - _handle_ts_search delegates the first cold call to a warm ts _daemon-serve (~130ms) instead of the ~5.7s in-process Nomic load; opt-in via TS_SEARCH_COLD_DELEGATE=1, falls back to in-process on any daemon failure - cli._daemon_serve routes ts_search through its handler (was 'unknown tool') 1779 tests pass (+8 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…4.0) The hardcoded string sat at 3.4.0 through the entire v4.x line, which masked that the daily-driver venv was running a stale build. Now sourced from importlib.metadata so it tracks pyproject.toml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mon delegation - scripts/ts_audit.py: automates the usage audit (latency, chains, adoption gaps, nudge fires, ML liveness); baseline snapshot for measuring nudge effect - telemetry.record_nudge/nudge_counts: persist per-kind nudge fires so effectiveness can be tracked vs adoption over successive audits - TCA revival: flush_session was never called (session_count stuck at 0 all deployment) -> now flushed at switch_project + atexit. Broken feature, not dead code; made live instead of removed - ts-daemon.service: launch venv python (has fastembed) so delegated ts_search is embedding-quality warm (~23ms) instead of substring 1786 tests pass (+7 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- server_handlers/resources.py: list + read observations as MCP resources (bounded, ranked by memory_index score) - wired in main() via list_resources/read_resource, opt-out TS_RESOURCES_DISABLED=1 - read-only, additive; tool path untouched 1792 tests pass (+6 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a regex-based .sh/.bash annotator (function declarations, source imports, doc comments, intra-file call graph) registered end-to-end: extension map, annotator map, default include patterns, pyproject keyword.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #51.
Adds a regex-based POSIX sh / bash annotator in the style of
go_annotator.py, registered end-to-end:.sh/.bashin_EXTENSION_MAP,annotate_shellin_ANNOTATOR_MAP,**/*.sh+**/*.bashin the default include patterns,shellin the pyproject keywords.What it extracts
name() {,name () {,function name {,function name() {), with brace-matched line ranges; a single-pass sanitizer blanks#comments, quoted-string interiors and heredoc bodies before brace counting, soawk '{ print $1 }', quoted braces and heredoc content never corrupt ranges or false-positive as declarations..andsource) asImportInfo(quoted/variable paths captured verbatim).#lines above a declaration) as the docstring.build_dependency_graph, with a shell keyword/builtin stoplist.Documented best-effort limits (each pinned by a test): dash-named bash functions (
my-func()) are skipped by the[A-Za-z_]\w*name grammar; subshell-body functions (name() ( … )) are captured with the declaration line only; the quote sanitizer is per-line, so a multi-line quoted string containing a declaration at line start can still false-positive (same class of limit as the other regex annotators).Tests
tests/test_markup_shell.py: 35 tests — all four declaration spellings, one-liners, nested-brace bodies with${var:-default}, doc comments, imports, dependency graph, and a hostile-input class (quoted/heredoc/comment declarations,casearms, awk braces,$(( )), truncated EOF, empty/whitespace input, shebang).Red→green: the two dispatch tests were executed on the untouched tree and failed with real assertions (
assert 0 == 1— generic annotator yields no functions), then passed with zero test edits after implementation + registration; also re-verified by reverting only the registration (both dispatch tests fail, rest of the file green).pytest tests/ -v: 1819 passed, 9 skipped, plus one failure intests/test_daemon_client.py::test_successful_call_returns_textthat is present on untouchedmainin my environment (macOS, Python 3.14) and unrelated.ruff check src/ tests/: clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01HTosXpNzKNTxHyY9gwkRKZ