diff --git a/docs/design/0005-node-type-registry.md b/docs/design/0005-node-type-registry.md new file mode 100644 index 00000000..1c854d7c --- /dev/null +++ b/docs/design/0005-node-type-registry.md @@ -0,0 +1,176 @@ +# Design Doc: YAML Node-Type Registry (Issue #43 Approach 2 Stepping Stone) + +> **Status:** Accepted +> **Date:** 2026-07-03 +> **Author:** Wolfvin +> **Related issues:** #43 +> **Related PRs:** (this PR) + +--- + +## Problem + +CodeLens has 7 tree-sitter parsers (Python, Rust, JS, TS, TSX, CSS, HTML). +Each parser hardcodes tree-sitter node-type strings like +`"function_definition"`, `"call_expression"`, `"import_statement"` directly +in Python code. Adding a new language means writing a ~250 LOC Python module +that hardcodes these strings — there's no config-driven path. + +Issue #43 proposed three approaches to make node discovery more declarative. +The full `.scm` query-file migration (Approach 1) was deferred pending +trigger conditions. This design doc covers **Approach 2 — the YAML node-type +registry** — which the issue explicitly recommends as a low-risk stepping +stone: "DO set up the YAML node-type registry as a low-risk stepping stone." + +## Goal + +Externalise hardcoded tree-sitter node-type lookups to a YAML config so +that adding a language's node-type mapping requires editing YAML, not +Python. Existing parsers continue to work unchanged — the YAML registry +is an additive new path that future parsers and the eventual `.scm` +engine can use. + +## Changes + +### Architecture / Data Model + +A new `scripts/languages/` package holds: + +``` +scripts/languages/ +├── __init__.py Re-exports public API +├── node_types.yaml Language → category → [node types] config +└── loader.py Cached YAML loader + public API +``` + +The YAML maps **semantic categories** (stable across languages) to +**concrete tree-sitter node types** (language-specific): + +```yaml +python: + function_def: [function_definition] + call: [call] + import: [import_statement] + +rust: + function_def: [function_item] + call: [call_expression] + import: [use_declaration] +``` + +The category names (`function_def`, `call`, `import`, `class_def`) are +the **same across all languages**. This is the design property that the +future `.scm` engine (issue #43 Phase B) will exploit: a `.scm` query +file for one language maps 1:1 to a category here. + +### New Files + +- `scripts/languages/__init__.py` — package marker, re-exports public API +- `scripts/languages/node_types.yaml` — config for all 7 tree-sitter languages +- `scripts/languages/loader.py` — `get_node_types()`, `get_language_config()`, + `get_supported_languages()`, `NodeTypeError`, cached frozenset results +- `tests/test_node_types.py` — 32 tests (28 pass, 4 skip without tree-sitter) +- `docs/design/0005-node-type-registry.md` — this design doc + +### Modified Files + +- `scripts/base_parser.py` — added `find_nodes_by_category(root, language, category)` + method (additive, backward compatible). Existing `find_nodes_by_type` and + `find_nodes_by_types` are unchanged. + +### CLI / MCP Surface + +No new command. No new MCP tool. This is an internal infrastructure change. + +### Tests + +- `tests/test_node_types.py` — covers: + - `get_node_types` for all 7 languages (returns correct frozensets) + - `get_language_config` (full category mapping, returns a copy) + - `get_supported_languages` (lists all 7, sorted) + - YAML config validity (no empty lists, no duplicates, all strings non-empty) + - Cache behaviour (frozenset identity, invalidation) + - Error handling (unknown language, unknown category, missing PyYAML) + - `BaseParser.find_nodes_by_category` integration (skipped when tree-sitter missing) + +## Trade-offs + +### Alternative A: Full `.scm` query-file migration (Approach 1) + +- **Pros:** Replaces ~250 LOC Python per language with ~50 LOC `.scm`. + One generic engine serves all languages. Portable to a future Rust core. +- **Cons:** 2-3 week migration effort. Only replaces Layer 1 (node + discovery) — Layer 2 (framework semantics) and Layer 3 (edge resolver) + stay in Python. Issue #43 says "Do not migrate existing parsers now" + until trigger conditions are met. +- **Why rejected (for now):** Trigger conditions were partially met (P0 + bugs #31/#32 closed, Phase 2 #22 clarified as deferred), but the issue + recommends the YAML registry as a lower-risk stepping stone first. The + `.scm` pilot is documented as Phase B (next step) in this design doc. + +### Alternative B: Hardcode node types in Python dicts (status quo) + +- **Pros:** Zero new files, zero new abstractions. +- **Cons:** Adding a language requires editing Python. No config-driven + path for future `.scm` migration. Node-type strings scattered across + 7 parser files with no single source of truth. +- **Why rejected:** Issue #43 explicitly recommends the YAML registry as + a "DO" action. The status quo doesn't prepare the ground for `.scm` + migration. + +### Chosen approach: YAML node-type registry (Approach 2) + +- **Why:** Lowest risk (zero migration, purely additive). Creates the + category abstraction that `.scm` will reuse. Adding a language = adding + YAML lines, not Python. The YAML is language-agnostic (reusable by a + future Rust core). Aligns with CodeLens's existing config patterns + (`plugin.yaml`, `hooks.json`, `codelens.config.json`). + +## Open Questions + +- [ ] Q1: Should existing parsers be refactored to use `find_nodes_by_category` + instead of `find_nodes_by_type`? (Owner: BOS, decide after this PR merges) + — This PR deliberately does NOT refactor existing parsers. The new method + is available for new parsers and the `.scm` pilot. Refactoring existing + parsers is Priority 6 in issue #43 ("2-3 weeks, Low ROI"). +- [ ] Q2: Should the YAML registry be extensible via plugins (like + `plugin.yaml`)? (Owner: BOS) — Out of scope for this PR. The plugin + angle is mentioned in issue #43 as a "future direction, not part of + this proposal." + +## Migration / Rollout + +No migration impact — additive change. Existing parsers continue to use +`find_nodes_by_type` / `find_nodes_by_types` unchanged. The new +`find_nodes_by_category` method is available for opt-in use. + +## Phase B: `.scm` Pilot (Next Step) + +This PR implements Phase A (YAML registry). Phase B — the Rust `.scm` +pilot described in issue #43 — is the recommended next step: + +1. Write `scripts/parsers/scm/rust.scm` covering function/method defs, + struct/enum/impl blocks, call sites, `use` imports, `#[attribute]`. +2. Build `scripts/parsers/scm_engine.py` — a generic engine that loads + `.scm` files, executes tree-sitter queries, returns nodes/edges in + the same shape as existing parsers. +3. Run both parsers (legacy + scm) on Rust test fixtures. Assert 100% + parity on captured nodes/edges. +4. Wire `scm_engine` into `scan.py` for Rust only, behind `--use-scm rust`. +5. If bake is clean: migrate Python, JS, TS, HTML, CSS one language per PR. +6. **Never migrate** Vue/Svelte/Blade — they have heavy Layer 2 framework + logic that `.scm` cannot express. + +Phase B requires tree-sitter's `Query` API which is not available in all +environments. The YAML registry (Phase A) works without tree-sitter at +runtime — it's pure config. + +## References + +- Issue: #43 +- Related issues: #46 (Semgrep-compat rule engine — closed, dependency met), + #22 (Phase 2 Rust core — closed as deferred) +- Prior art: [ast-grep](https://ast-grep.github.io/), + [tree-sitter query syntax](https://tree-sitter.github.io/tree-sitter/using-parsers#query-syntax), + [Semgrep rule syntax](https://semgrep.dev/docs/writing-rules/overview/) +- Related design docs: [0001-taint-engine](0001-taint-engine.md) diff --git a/scripts/base_parser.py b/scripts/base_parser.py index 6d1444aa..bad6c835 100755 --- a/scripts/base_parser.py +++ b/scripts/base_parser.py @@ -206,6 +206,57 @@ def visitor(node, source, depth): self.walk_tree(root, b'', visitor) return results + # ─── Issue #43: YAML node-type registry (Approach 2 stepping stone) ── + # + # ``find_nodes_by_category`` is the config-driven counterpart of + # ``find_nodes_by_types``. Instead of passing a hardcoded set of + # node-type strings, the caller passes a language + semantic category + # (e.g. ``"python"``, ``"function_def"``) and the method looks up the + # concrete node types from ``scripts/languages/node_types.yaml``. + # + # This is purely additive — existing parsers continue to use + # ``find_nodes_by_type`` / ``find_nodes_by_types`` unchanged. New + # parsers and the future .scm engine (issue #43 Phase B) can use + # ``find_nodes_by_category`` to avoid hardcoding node types. + # + # Why a method on BaseParser (not a free function)? + # ``find_nodes_by_types`` is already a method here and uses + # ``self.walk_tree``. Keeping the category-based variant next to + # it makes the API discoverable and lets subclasses override + # both with the same pattern. + + def find_nodes_by_category(self, root: Node, language: str, category: str) -> List[Node]: + """Find all nodes matching a semantic category from the YAML registry. + + Args: + root: The AST root node to walk. + language: Language key in the YAML registry (e.g. ``"python"``, + ``"rust"``, ``"javascript"``). + category: Semantic category (e.g. ``"function_def"``, ``"call"``, + ``"import"``). + + Returns: + List of nodes whose ``type`` matches any of the concrete node + types registered for ``language`` + ``category``. + + Raises: + NodeTypeError: If the language or category is not in the YAML + registry. The error is allowed to propagate because it + signals a config bug, not a parse error — the caller + should fix the config, not catch the exception. + + Example:: + + parser = PythonParser(language) + root = parser.parse(source) + # Instead of: parser.find_nodes_by_type(root, "function_definition") + # Use: parser.find_nodes_by_category(root, "python", "function_def") + funcs = parser.find_nodes_by_category(root, "python", "function_def") + """ + from languages import get_node_types + node_types = get_node_types(language, category) + return self.find_nodes_by_types(root, node_types) + def find_parent_of_type(self, node: Node, parent_type: str) -> Optional[Node]: """Walk up the tree to find a parent of a specific type.""" current = node.parent diff --git a/scripts/languages/__init__.py b/scripts/languages/__init__.py new file mode 100644 index 00000000..272abd5f --- /dev/null +++ b/scripts/languages/__init__.py @@ -0,0 +1,45 @@ +# @WHO: scripts/languages/__init__.py +# @WHAT: Language config package — YAML node-type registry for tree-sitter parsers (issue #43) +# @PART: languages +# @ENTRY: - +# +# Issue #43 — Approach 2 (YAML node-type registry) stepping stone. +# +# This package externalises the hardcoded tree-sitter node-type lookups +# (``find_nodes_by_type(root, "function_definition")``) to a YAML config. +# The Python walker stays, but reads node types from config instead of +# hardcoding them. Adding a language = adding a few lines of YAML. +# +# This is the low-risk stepping stone recommended by issue #43 before +# any .scm migration. It does NOT change existing parser behaviour — +# it adds a new config-driven path (``find_nodes_by_category``) that +# future parsers and the eventual .scm engine can use. + +"""Language config package — YAML node-type registry. + +Public entry points:: + + from languages import get_node_types, get_language_config + + types = get_node_types("python", "function_def") + # → frozenset({"function_definition"}) + +The registry maps ``language → category → [tree-sitter node types]``. +Categories are semantic (``function_def``, ``call``, ``import``, etc.) +so the same category name works across languages — this is the design +property that the future .scm engine (issue #43 Phase B) will exploit. +""" + +from .loader import ( # noqa: F401 + get_node_types, + get_language_config, + get_supported_languages, + NodeTypeError, +) + +__all__ = [ + "get_node_types", + "get_language_config", + "get_supported_languages", + "NodeTypeError", +] diff --git a/scripts/languages/loader.py b/scripts/languages/loader.py new file mode 100644 index 00000000..8a5920de --- /dev/null +++ b/scripts/languages/loader.py @@ -0,0 +1,242 @@ +# @WHO: scripts/languages/loader.py +# @WHAT: YAML node-type registry loader — cached access to language node-type config (issue #43) +# @PART: languages +# @ENTRY: get_node_types(), get_language_config() +# +# @FLOW: NODE_TYPE_LOOKUP +# @CALLS: yaml.safe_load() -> dict, internal _load_registry() -> cached dict +# @MUTATES: none (pure read; cache is module-level but effectively immutable after first load) +# +# Issue #43 — Approach 2 (YAML node-type registry) loader. +# +# Design: +# - The YAML file is loaded once on first access and cached in a module-level +# dict. Subsequent calls are O(1) dict lookups — no file I/O. +# - The cache is intentionally NOT thread-safe with a lock. The worst case +# under concurrent first-access is that two threads both load the YAML +# (idempotent — same data) and one overwrites the other's cache entry +# (same data again). No corruption, no race condition that matters. +# - ``get_node_types`` returns a ``frozenset`` (not a list) so callers can +# do ``if node.type in types`` without worrying about duplicates or order. +# - Unknown language or category raises ``NodeTypeError`` — fail loud, not +# silent. Silent fallback to an empty set would hide config typos and +# produce empty parse results that are hard to debug. + +"""YAML node-type registry loader. + +Public API:: + + from languages import get_node_types + + types = get_node_types("python", "function_def") + # → frozenset({"function_definition"}) + + types = get_node_types("rust", "call") + # → frozenset({"call_expression"}) + + config = get_language_config("python") + # → {"function_def": [...], "class_def": [...], ...} + + langs = get_supported_languages() + # → ["css", "html", "javascript", "python", "rust", "tsx", "typescript"] +""" + +from __future__ import annotations + +import os +from typing import Any, Dict, FrozenSet, List, Optional + +# Lazy import — PyYAML is an optional dep in minimal installs. The import +# error surfaces as a clear ``NodeTypeError`` on first use rather than at +# module import time, so ``from languages import ...`` never crashes the +# CLI even when PyYAML is missing. +try: + import yaml + _YAML_AVAILABLE = True +except ImportError: + _YAML_AVAILABLE = False + + +class NodeTypeError(KeyError): + """Raised when a language or category is not in the registry. + + Subclasses ``KeyError`` so ``except KeyError`` still catches it, but + the explicit subclass lets callers distinguish "config typo" from + "genuine missing key" if they need to. + """ + + +# ─── Module-level cache ──────────────────────────────────────────────────── + +# Cached registry: ``{language: {category: [node_types]}}``. +# ``None`` means "not yet loaded". After first load, always a dict. +_registry: Optional[Dict[str, Dict[str, List[str]]]] = None + +# Cached frozensets per (language, category) pair. ``get_node_types`` +# returns these directly so callers don't recreate frozensets on every +# call. Keyed by ``f"{language}:{category}"``. +_frozenset_cache: Dict[str, FrozenSet[str]] = {} + +# Path to the YAML file, resolved relative to this module. +_YAML_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "node_types.yaml") + + +def _load_registry() -> Dict[str, Dict[str, List[str]]]: + """Load the YAML registry from disk (cached after first call). + + Returns the full registry dict. On first call, reads the YAML file; + subsequent calls return the cached dict. + + Raises: + NodeTypeError: If PyYAML is not installed or the YAML file + cannot be read/parsed. The error message includes the + file path and the underlying cause for debugging. + """ + global _registry + if _registry is not None: + return _registry + + if not _YAML_AVAILABLE: + raise NodeTypeError( + "[languages.loader] PyYAML not installed — cannot load node-type registry. " + "Install with: pip install pyyaml" + ) + + try: + with open(_YAML_PATH, "r", encoding="utf-8") as f: + data = yaml.safe_load(f) + except OSError as exc: + raise NodeTypeError( + f"[languages.loader] cannot read node-type registry at {_YAML_PATH}: {exc}" + ) from exc + except yaml.YAMLError as exc: + raise NodeTypeError( + f"[languages.loader] YAML parse error in {_YAML_PATH}: {exc}" + ) from exc + + if not isinstance(data, dict): + raise NodeTypeError( + f"[languages.loader] expected top-level dict in {_YAML_PATH}, got {type(data).__name__}" + ) + + # Validate structure: each language must map to a dict of category → list. + validated: Dict[str, Dict[str, List[str]]] = {} + for lang, categories in data.items(): + if not isinstance(categories, dict): + raise NodeTypeError( + f"[languages.loader] language {lang!r} must map to a dict of " + f"category → [node_types], got {type(categories).__name__}" + ) + validated[lang] = {} + for cat, types in categories.items(): + if not isinstance(types, list): + raise NodeTypeError( + f"[languages.loader] language {lang!r} category {cat!r} must be a " + f"list of node-type strings, got {type(types).__name__}" + ) + for t in types: + if not isinstance(t, str): + raise NodeTypeError( + f"[languages.loader] language {lang!r} category {cat!r} contains " + f"non-string node type: {t!r}" + ) + validated[lang][cat] = types + + _registry = validated + return _registry + + +def _invalidate_cache() -> None: + """Drop the cached registry and frozenset cache. Used by tests.""" + global _registry + _registry = None + _frozenset_cache.clear() + + +# ─── Public API ──────────────────────────────────────────────────────────── + + +def get_node_types(language: str, category: str) -> FrozenSet[str]: + """Return the set of tree-sitter node types for ``language`` + ``category``. + + Args: + language: Language key in the YAML (e.g. ``"python"``, ``"rust"``, + ``"javascript"``, ``"typescript"``, ``"tsx"``, ``"css"``, + ``"html"``). Case-sensitive — must match the YAML key exactly. + category: Semantic category (e.g. ``"function_def"``, ``"call"``, + ``"import"``, ``"class_def"``). Case-sensitive. + + Returns: + ``frozenset`` of node-type strings. The set is frozen so callers + can't accidentally mutate the cached data. + + Raises: + NodeTypeError: If ``language`` or ``category`` is not in the + registry. Fail loud — a config typo should surface immediately, + not silently produce empty parse results. + + Example:: + + >>> get_node_types("python", "function_def") + frozenset({'function_definition'}) + >>> get_node_types("rust", "call") + frozenset({'call_expression'}) + >>> get_node_types("javascript", "call") + frozenset({'call_expression', 'new_expression'}) + """ + registry = _load_registry() + cache_key = f"{language}:{category}" + cached = _frozenset_cache.get(cache_key) + if cached is not None: + return cached + + lang_config = registry.get(language) + if lang_config is None: + raise NodeTypeError( + f"[languages.loader] language {language!r} not in registry. " + f"Supported: {sorted(registry.keys())}" + ) + types = lang_config.get(category) + if types is None: + raise NodeTypeError( + f"[languages.loader] category {category!r} not defined for language " + f"{language!r}. Available categories: {sorted(lang_config.keys())}" + ) + result = frozenset(types) + _frozenset_cache[cache_key] = result + return result + + +def get_language_config(language: str) -> Dict[str, List[str]]: + """Return the full category → node-types mapping for one language. + + Returns a shallow copy so callers can't mutate the cached registry. + + Raises: + NodeTypeError: If ``language`` is not in the registry. + """ + registry = _load_registry() + lang_config = registry.get(language) + if lang_config is None: + raise NodeTypeError( + f"[languages.loader] language {language!r} not in registry. " + f"Supported: {sorted(registry.keys())}" + ) + # Shallow copy — the lists inside are also copied so callers can't + # mutate the cached lists either. + return {cat: list(types) for cat, types in lang_config.items()} + + +def get_supported_languages() -> List[str]: + """Return a sorted list of all languages in the registry.""" + registry = _load_registry() + return sorted(registry.keys()) + + +__all__ = [ + "get_node_types", + "get_language_config", + "get_supported_languages", + "NodeTypeError", + "_invalidate_cache", +] diff --git a/scripts/languages/node_types.yaml b/scripts/languages/node_types.yaml new file mode 100644 index 00000000..fd152610 --- /dev/null +++ b/scripts/languages/node_types.yaml @@ -0,0 +1,158 @@ +# @WHO: scripts/languages/node_types.yaml +# @WHAT: Tree-sitter node-type registry — maps language+category to concrete AST node types (issue #43) +# @PART: languages +# @ENTRY: - +# +# Issue #43 — Approach 2 (YAML node-type registry). +# +# Each language maps semantic categories (function_def, call, import, etc.) +# to concrete tree-sitter node types. The categories are stable across +# languages — ``function_def`` means "function definition" whether the +# language is Python, Rust, or JS. This is the design property that the +# future .scm engine (issue #43 Phase B) will exploit: a .scm query file +# for one language maps 1:1 to a category here. +# +# Adding a new tree-sitter language: +# 1. Add an entry here with the language's node types. +# 2. The loader picks it up automatically (no code change). +# 3. Future parsers can use ``find_nodes_by_category(root, lang, cat)``. +# +# Updating an existing language: +# - Just edit the YAML. No code change, no test change (unless the +# test asserts specific node types). +# +# Why YAML and not Python dicts? +# - Non-contributors can add a language without touching Python. +# - The same config is reusable by a future Rust core (PyO3) — YAML +# is language-agnostic. +# - Aligns with CodeLens's existing config patterns (codelens.config.json, +# plugin.yaml, hooks.json). + +# ─── Python ────────────────────────────────────────────────────────── +python: + function_def: + - function_definition + class_def: + - class_definition + call: + - call + import: + - import_statement + decorator: + - decorator + # Identifier sub-types used for call-target resolution + identifier: + - identifier + - attribute + +# ─── Rust ──────────────────────────────────────────────────────────── +rust: + function_def: + - function_item + struct: + - struct_item + enum: + - enum_item + trait: + - trait_item + module: + - mod_item + const: + - const_item + - static_item + type_alias: + - type_item + macro: + - macro_definition + import: + - use_declaration + impl: + - impl_item + call: + - call_expression + # Identifier sub-types used for call-target resolution + identifier: + - identifier + - field_expression + - scoped_identifier + +# ─── JavaScript (backend / Node.js) ───────────────────────────────── +javascript: + function_def: + - function_declaration + - generator_function_declaration + variable_declarator: + - variable_declarator + class_def: + - class_declaration + export: + - export_statement + import: + - import_statement + call: + - call_expression + - new_expression + identifier: + - identifier + - member_expression + +# ─── TypeScript (backend) ─────────────────────────────────────────── +typescript: + function_def: + - function_declaration + - generator_function_declaration + variable_declarator: + - variable_declarator + class_def: + - class_declaration + export: + - export_statement + import: + - import_statement + call: + - call_expression + - new_expression + identifier: + - identifier + - member_expression + +# ─── TSX (React) ──────────────────────────────────────────────────── +tsx: + function_def: + - function_declaration + variable_declarator: + - variable_declarator + class_def: + - class_declaration + export: + - export_statement + import: + - import_statement + call: + - call_expression + - new_expression + jsx_element: + - jsx_opening_element + - jsx_self_closing_element + jsx_attribute: + - jsx_attribute + identifier: + - identifier + - member_expression + - nested_identifier + +# ─── CSS ──────────────────────────────────────────────────────────── +css: + keyframes: + - keyframes_statement + class_selector: + - class_selector + id_selector: + - id_selector + pseudo_class_selector: + - pseudo_class_selector + +# ─── HTML ─────────────────────────────────────────────────────────── +html: + attribute: + - attribute diff --git a/tests/test_node_types.py b/tests/test_node_types.py new file mode 100644 index 00000000..ac611568 --- /dev/null +++ b/tests/test_node_types.py @@ -0,0 +1,404 @@ +"""Tests for the YAML node-type registry (issue #43 — Approach 2 stepping stone). + +Scope: + +* :mod:`languages.loader` — :func:`get_node_types`, :func:`get_language_config`, + :func:`get_supported_languages`, :class:`NodeTypeError`. +* :mod:`languages.node_types` (YAML) — validates that all 7 tree-sitter + languages are present with sensible categories. +* :class:`base_parser.BaseParser.find_nodes_by_category` — verifies the + config-driven lookup path (without needing tree-sitter installed; the + test mocks the tree walk). + +All tests are network-free and filesystem-light — they read the bundled +``node_types.yaml`` directly. No tree-sitter installation is required. +""" + +from __future__ import annotations + +import os +import sys +from typing import Any, List +from unittest import mock + +import pytest + +# ─── Path setup (mirror other tests) ─────────────────────────────────────── + +_THIS_DIR = os.path.dirname(os.path.abspath(__file__)) +_SCRIPTS_DIR = os.path.join(os.path.dirname(_THIS_DIR), "scripts") +if _SCRIPTS_DIR not in sys.path: + sys.path.insert(0, _SCRIPTS_DIR) + +from languages import ( # noqa: E402 + get_node_types, + get_language_config, + get_supported_languages, + NodeTypeError, +) +from languages import loader as loader_mod # noqa: E402 + + +# ─── Fixtures ────────────────────────────────────────────────────────────── + + +@pytest.fixture(autouse=True) +def reset_cache(): + """Drop the loader's module-level cache before and after each test. + + Without this, a test that mocks ``yaml.safe_load`` would be ignored + because the cache already holds the real data from a prior test. + """ + loader_mod._invalidate_cache() + yield + loader_mod._invalidate_cache() + + +# ─── get_node_types ──────────────────────────────────────────────────────── + + +class TestGetNodeTypes: + """``get_node_types(language, category)`` returns the right frozenset.""" + + def test_python_function_def(self): + result = get_node_types("python", "function_def") + assert result == frozenset({"function_definition"}) + + def test_python_call(self): + result = get_node_types("python", "call") + assert result == frozenset({"call"}) + + def test_rust_call(self): + result = get_node_types("rust", "call") + assert result == frozenset({"call_expression"}) + + def test_javascript_call_has_multiple_types(self): + """JS calls include both ``call_expression`` and ``new_expression``.""" + result = get_node_types("javascript", "call") + assert "call_expression" in result + assert "new_expression" in result + assert len(result) == 2 + + def test_rust_const_has_multiple_types(self): + """Rust const category includes both ``const_item`` and ``static_item``.""" + result = get_node_types("rust", "const") + assert "const_item" in result + assert "static_item" in result + + def test_returns_frozenset(self): + """Result must be a frozenset so callers can't mutate the cache.""" + result = get_node_types("python", "function_def") + assert isinstance(result, frozenset) + + def test_unknown_language_raises(self): + with pytest.raises(NodeTypeError, match="not in registry"): + get_node_types("cobol", "function_def") + + def test_unknown_category_raises(self): + with pytest.raises(NodeTypeError, match="not defined for language"): + get_node_types("python", "nonexistent_category") + + def test_error_message_lists_supported_languages(self): + """The error message should list supported languages for discoverability.""" + try: + get_node_types("cobol", "function_def") + except NodeTypeError as e: + msg = str(e) + assert "python" in msg.lower() + assert "rust" in msg.lower() + + def test_error_message_lists_available_categories(self): + """The error message should list available categories for the language.""" + try: + get_node_types("python", "nonexistent") + except NodeTypeError as e: + msg = str(e) + assert "function_def" in msg + assert "class_def" in msg + + +# ─── get_language_config ────────────────────────────────────────────────── + + +class TestGetLanguageConfig: + """``get_language_config(language)`` returns the full category mapping.""" + + def test_python_config_has_expected_categories(self): + config = get_language_config("python") + assert "function_def" in config + assert "class_def" in config + assert "call" in config + assert "import" in config + + def test_rust_config_has_expected_categories(self): + config = get_language_config("rust") + for cat in ("function_def", "struct", "enum", "trait", "impl", + "call", "import", "macro"): + assert cat in config, f"rust config missing category: {cat}" + + def test_config_values_are_lists(self): + config = get_language_config("python") + for cat, types in config.items(): + assert isinstance(types, list), f"{cat} value is not a list" + assert all(isinstance(t, str) for t in types), \ + f"{cat} contains non-string type" + + def test_config_is_a_copy(self): + """Modifying the returned config must not affect the cached registry.""" + config1 = get_language_config("python") + config1["function_def"].append("FAKE_TYPE") + config1["injected"] = ["bad"] + + config2 = get_language_config("python") + assert "FAKE_TYPE" not in config2["function_def"] + assert "injected" not in config2 + + def test_unknown_language_raises(self): + with pytest.raises(NodeTypeError): + get_language_config("cobol") + + +# ─── get_supported_languages ────────────────────────────────────────────── + + +class TestGetSupportedLanguages: + """``get_supported_languages()`` lists all 7 tree-sitter languages.""" + + EXPECTED_LANGUAGES = { + "python", "rust", "javascript", "typescript", "tsx", "css", "html" + } + + def test_all_expected_languages_present(self): + langs = set(get_supported_languages()) + missing = self.EXPECTED_LANGUAGES - langs + assert not missing, f"missing languages in registry: {missing}" + + def test_returns_sorted_list(self): + langs = get_supported_languages() + assert langs == sorted(langs) + + def test_no_extra_languages_beyond_expected(self): + """The registry should only contain the 7 tree-sitter languages. + + If a new language is added, this test will flag it so the test + suite stays in sync with the config. + """ + langs = set(get_supported_languages()) + extra = langs - self.EXPECTED_LANGUAGES + # Allow extra languages but warn — this is a soft assertion. + # If you add a language, update EXPECTED_LANGUAGES. + if extra: + pytest.fail( + f"Unexpected languages in registry: {extra}. " + f"Update EXPECTED_LANGUAGES in this test." + ) + + +# ─── YAML config validity ───────────────────────────────────────────────── + + +class TestYamlConfigValidity: + """Validate the structure and content of node_types.yaml.""" + + def test_every_language_has_function_def_or_equivalent(self): + """Every language that has functions should have a function_def category. + + CSS and HTML don't have functions, so they're excluded. + """ + langs_with_functions = {"python", "rust", "javascript", "typescript", "tsx"} + for lang in langs_with_functions: + config = get_language_config(lang) + assert "function_def" in config, \ + f"{lang} should have a function_def category" + + def test_every_language_has_call_or_equivalent(self): + """Languages with function calls should have a call category.""" + langs_with_calls = {"python", "rust", "javascript", "typescript", "tsx"} + for lang in langs_with_calls: + config = get_language_config(lang) + assert "call" in config, \ + f"{lang} should have a call category" + + def test_no_empty_category_lists(self): + """No category should have an empty node-type list.""" + for lang in get_supported_languages(): + config = get_language_config(lang) + for cat, types in config.items(): + assert len(types) > 0, \ + f"{lang}.{cat} has an empty node-type list" + + def test_no_duplicate_node_types_within_category(self): + """No category should list the same node type twice.""" + for lang in get_supported_languages(): + config = get_language_config(lang) + for cat, types in config.items(): + assert len(types) == len(set(types)), \ + f"{lang}.{cat} has duplicate node types: {types}" + + def test_all_node_types_are_nonempty_strings(self): + """Every node type must be a non-empty string.""" + for lang in get_supported_languages(): + config = get_language_config(lang) + for cat, types in config.items(): + for t in types: + assert isinstance(t, str) and len(t) > 0, \ + f"{lang}.{cat} has invalid node type: {t!r}" + + +# ─── Cache behaviour ─────────────────────────────────────────────────────── + + +class TestCacheBehaviour: + """The module-level cache should prevent repeated file reads.""" + + def test_cache_returns_same_object_for_same_language(self): + """Two calls for the same language should return the same cached frozenset. + + This is an implementation detail, but it verifies that the cache + is working (no repeated YAML parsing). + """ + result1 = get_node_types("python", "function_def") + result2 = get_node_types("python", "function_def") + assert result1 is result2 + + def test_invalidate_cache_forces_reload(self): + """``_invalidate_cache`` forces the next call to re-read the YAML.""" + result1 = get_node_types("python", "function_def") + loader_mod._invalidate_cache() + result2 = get_node_types("python", "function_def") + # Same value, but the cache was rebuilt — can't assert identity + # because frozenset() creates a new object. Just verify equality. + assert result1 == result2 + + +# ─── Error handling: missing PyYAML ──────────────────────────────────────── + + +class TestMissingPyYAML: + """When PyYAML is not installed, the loader should fail clearly.""" + + def test_missing_yaml_raises_nodetypeerror(self, monkeypatch): + """Simulate PyYAML not being installed.""" + monkeypatch.setattr(loader_mod, "_YAML_AVAILABLE", False) + loader_mod._invalidate_cache() + with pytest.raises(NodeTypeError, match="PyYAML not installed"): + get_node_types("python", "function_def") + + +# ─── BaseParser.find_nodes_by_category integration ──────────────────────── + + +# tree_sitter is an optional dependency. The BaseParser tests below need +# it because base_parser.py imports from tree_sitter at module level. +# Tests for the loader itself (above) don't need tree_sitter and always run. +_tree_sitter_available = True +try: + import tree_sitter # noqa: F401 +except ImportError: + _tree_sitter_available = False + +_BASEPARSER_SKIP_REASON = "tree_sitter not installed — BaseParser tests require it" + + +@pytest.mark.skipif(not _tree_sitter_available, reason=_BASEPARSER_SKIP_REASON) +class TestBaseParserFindByCategory: + """Verify ``BaseParser.find_nodes_by_category`` wires the YAML lookup + into the existing ``find_nodes_by_types`` tree walk. + + These tests mock ``find_nodes_by_types`` so they don't need a real + tree-sitter AST — they verify the wiring, not the walk. + """ + + def test_find_nodes_by_category_calls_find_nodes_by_types(self): + """``find_nodes_by_category`` should delegate to ``find_nodes_by_types`` + with the YAML-resolved node-type set.""" + # We can't construct a real BaseParser without a tree-sitter Language, + # so we create a bare instance via __new__ and mock the methods. + from base_parser import BaseParser + + parser = BaseParser.__new__(BaseParser) + + # Mock find_nodes_by_types to capture what it's called with. + captured_types = None + + def mock_find(root, node_types): + nonlocal captured_types + captured_types = node_types + return ["mock_node"] + + parser.find_nodes_by_types = mock_find + + result = parser.find_nodes_by_category( + root="fake_root", + language="python", + category="function_def", + ) + + assert result == ["mock_node"] + assert captured_types == frozenset({"function_definition"}) + + def test_find_nodes_by_category_multi_type(self): + """Categories with multiple node types pass the full set through.""" + from base_parser import BaseParser + + parser = BaseParser.__new__(BaseParser) + captured_types = None + + def mock_find(root, node_types): + nonlocal captured_types + captured_types = node_types + return [] + + parser.find_nodes_by_types = mock_find + + parser.find_nodes_by_category( + root="fake_root", + language="javascript", + category="call", + ) + + assert captured_types == frozenset({"call_expression", "new_expression"}) + + def test_find_nodes_by_category_unknown_language_propagates(self): + """An unknown language should raise NodeTypeError, not silently return [].""" + from base_parser import BaseParser + + parser = BaseParser.__new__(BaseParser) + parser.find_nodes_by_types = lambda root, types: [] + + with pytest.raises(NodeTypeError): + parser.find_nodes_by_category( + root="fake_root", + language="cobol", + category="function_def", + ) + + def test_find_nodes_by_category_unknown_category_propagates(self): + """An unknown category should raise NodeTypeError, not silently return [].""" + from base_parser import BaseParser + + parser = BaseParser.__new__(BaseParser) + parser.find_nodes_by_types = lambda root, types: [] + + with pytest.raises(NodeTypeError): + parser.find_nodes_by_category( + root="fake_root", + language="python", + category="nonexistent", + ) + + +# ─── YAML file location ──────────────────────────────────────────────────── + + +class TestYamlFileLocation: + """Verify the loader finds the YAML file relative to itself.""" + + def test_yaml_file_exists(self): + """The YAML file should exist next to the loader module.""" + yaml_path = loader_mod._YAML_PATH + assert os.path.exists(yaml_path), f"node_types.yaml not found at {yaml_path}" + + def test_yaml_path_is_absolute(self): + """The path should be absolute so it works regardless of CWD.""" + assert os.path.isabs(loader_mod._YAML_PATH)