feat(search): semantic_query MCP tool via TF-IDF Option A (closes #11)

[Wolfvin]

Closes #11

Summary

Implements Option A from issue #11: a zero-dependency, pure-Python TF-IDF semantic search engine exposed as a new semantic-query CLI command and codelens_semantic_query MCP tool.

Agents can now call:

semantic_query(query="user authentication flow", top_k=10)

and get back ranked symbols by cosine similarity — so a query for "auth" surfaces verify_jwt_claims (matching via the auth/jwt.py file path) even when no symbol literally contains "auth" in its name.

What was added

File	Purpose
scripts/semantic_search_engine.py	TF-IDF engine: tokenizer (splits snake_case, camelCase, kebab-case, paths, signature punctuation), vocabulary + smoothed IDF, sparse document vectors, cosine-similarity ranking. Cached per (db_path, mtime) so a re-scan auto-invalidates the index.
scripts/commands/semantic_query.py	CLI command registered as semantic-query. Auto-exposed as codelens_semantic_query MCP tool via dynamic tool discovery.
scripts/mcp_server.py	Added explicit static tool definition in _TOOL_DEFINITIONS for documented JSON schema (query, workspace, top).
tests/test_semantic_search_engine.py	31 tests: tokenizer, end-to-end query, cache invalidation, graceful degradation, TF-IDF ranking signal.
README.md, SKILL.md, SKILL-QUICK.md, pyproject.toml, skill.json, scripts/graph_model.py	Auto-synced by sync_command_count.py --apply: CLI 68->70, MCP 66->68.

Naming note

The module is named semantic_search_engine.py (not semantic_engine.py) to avoid colliding with the pre-existing deprecated taint-analysis semantic_engine.py from PR #140 / issue #49 phase-1. The two modules share the "semantic" prefix but have no other relationship.

Why Option A first

Per the issue:

Recommend starting with Option A as non-breaking enhancement, Option B as optional install.

Option A (TF-IDF) is non-breaking, zero-dependency, deterministic, and good enough for the majority of agent "find the right file" queries. Option B (sentence-transformers embeddings) remains future work — it would require bundling an ~80 MB model and is better added as an optional install behind a feature flag.

Non-breaking by design

• New module + new command — no existing scan flow, schema, or output shape touched.
• Reads from the existing SQLite symbols table populated by persistent_registry — no new index file to keep consistent.
• All 1401 existing tests continue to pass (plus 31 new tests added).

Tests

PYTHONUTF8=1 PYTHONPATH=scripts python -m pytest tests/ --ignore=tests/test_integration.py

Result on this branch: 1401 passed, 80 skipped, 1 failed.

The 1 failure (test_codelensignore::TestBackwardCompat::test_actual_target_dir_is_ignored) is pre-existing and unrelated — it fails identically on clean origin/main (verified via git stash + re-run). Environmental issue with the target directory ignore pattern in the sandbox.

Limitations (documented, not bugs)

• No stemming. Pure TF-IDF does not stem, so "authentication" will not match "auth". The module docstring documents this and explains the trade-off (Option B would handle it). Workaround: agents query with "auth" directly.
• No docstring indexing. The current persistent_registry.symbols schema doesn't store docstrings, so the TF-IDF document is built from name + signature + kind + language + file_path + extra_json. Future enhancement: store docstrings during scan and add them to the document text.
• In-memory index. For very large codebases (>50k symbols) the in-memory TF-IDF matrix may use noticeable RAM. The cache is keyed by (db_path, mtime) so it's rebuilt only when the registry changes.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sonarqubecloud]

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE