diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6731e0b..aefbf14 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -136,6 +136,9 @@ jobs: ruff format --check app mypy app + - name: Validate capability registry and README + run: python scripts/check-capabilities.py + - name: Test working-directory: apps/backend env: diff --git a/README.md b/README.md index 1fe90b0..935e6c1 100644 --- a/README.md +++ b/README.md @@ -61,26 +61,28 @@ Across those surfaces, PARTHA keeps the repository revision, snapshot identity, Statuses describe executable behaviour on the current `dev` branch: -| Capability | Assessment | Current boundary | + +| Capability | Status | Current boundary | | --- | --- | --- | | Archive upload and public GitHub import | **Implemented** | ZIP/TAR-family archives and shallow public GitHub HTTPS clones; size and path-safety limits apply. Private GitHub cloning and other repository hosts are not supported. | | Repository explorer | **Implemented** | Owner-scoped file tree plus bounded text/image preview, binary detection, and truncation. | | Authentication and owner isolation | **Implemented** | Email/password, Argon2, short-lived access tokens, rotating refresh tokens with reuse detection. Protected resources are owner-scoped; non-owner access returns 404. | | Analysis lifecycle | **Implemented** | Database-backed, cancellable job with progress, bounded retry, lease renewal, and stale-worker recovery. | -| Repository Intelligence | **Implemented, limited** | Immutable, revision-addressed `ri.v1` snapshots with normalized facts, evidence, query APIs, and a total canonical graph hash. Semantic extraction is strongest for supported Python and TypeScript/JavaScript constructs. | -| Architecture and authentication explanation | **Implemented, limited** | Interactive snapshot-backed graph. Module/layer classification is heuristic. The cited authentication subgraph covers supported Python/FastAPI patterns only. | -| Dependency Graph | **Implemented, limited** | Direct declarations from `package.json`, `requirements.txt`, and `pyproject.toml`, including repeated workspace declarations and exact manifest spans. No lockfiles or transitive resolution. | -| Engineering Review | **Implemented, limited** | `engineering-review.v2`; evidence-addressed findings and explicit category states. No overall score, grade, health percentage, vulnerability result, or generated roadmap. | -| Repository Insights | **Implemented, limited** | `repository-insights.v1`; defined counts, ratios, diagnostics, language breakdowns, and extraction coverage from one snapshot. No change-over-time claims. | -| Documentation and report export | **Implemented, limited** | Documentation uses current-revision structural facts. Review, Documentation, Architecture, and Dependencies export through one JSON/Markdown/HTML/PDF pipeline. | -| AI provider integration | **Implemented, limited** | Per-user configuration for supported providers, encrypted API keys, and constrained outbound destinations. Free-form answers receive structural facts and observed paths—not source bytes or line spans—and return no automatic citations. | -| Asynchronous processing | **Partially implemented** | Analysis runs off the request path. Import, extraction of the initial archive/clone, and file-tree parsing remain synchronous; one in-process worker handles analysis jobs. | -| Incremental re-analysis and revision comparison | **Not implemented / not assessed** | The full repository is analysed again; no snapshot-to-snapshot product workflow is available. | -| Change-impact or blast-radius analysis | **Implemented, limited** | Owner-scoped traversal over one sealed snapshot's resolved import and dependency edges. It does not compare revisions or calculate churn or trends. | -| Vulnerability and outdated-dependency scanning | **Not implemented / not assessed** | Dependency responses report explicit `not_computed` states; Review keeps vulnerability scanning `not_assessed`. No clean bill of health or zero count is fabricated. | -| Grounded, cited free-form AI answers | **Not implemented / not assessed** | Provider answers are intentionally uncited because providers do not receive source content or line numbers. | - -**Implemented, limited** means the workflow exists but has a disclosed coverage or trust boundary. **Partially implemented** means only part of the end-to-end behaviour exists. **Not implemented / not assessed** means PARTHA does not manufacture an answer. +| Repository Intelligence | **Implemented with disclosed limits** | Immutable, revision-addressed `ri.v1` snapshots with normalized facts, evidence, query APIs, and a total canonical graph hash. Semantic extraction is strongest for supported Python and TypeScript/JavaScript constructs. | +| Architecture and authentication explanation | **Implemented with disclosed limits** | Interactive snapshot-backed graph. Module/layer classification is heuristic. The cited authentication subgraph covers supported Python/FastAPI patterns only. | +| Dependency Graph | **Implemented with disclosed limits** | Direct declarations from `package.json`, `pyproject.toml`, and `requirements.txt`, including repeated workspace declarations and exact manifest spans. No lockfiles or transitive resolution. | +| Engineering Review | **Implemented with disclosed limits** | `engineering-review.v2`; evidence-addressed findings and explicit category states. No overall score, grade, health percentage, vulnerability result, or generated roadmap. | +| Repository Insights | **Implemented with disclosed limits** | `repository-insights.v1`; defined counts, ratios, diagnostics, language breakdowns, and extraction coverage from one snapshot. No change-over-time claims. | +| Documentation and report export | **Implemented with disclosed limits** | Documentation uses current-revision structural facts. Review, Documentation, Architecture, and Dependencies export through one JSON/Markdown/HTML/PDF pipeline. | +| AI provider integration | **Implemented with disclosed limits** | Per-user configuration for supported providers, encrypted API keys, and constrained outbound destinations. Free-form answers receive structural facts and observed paths—not source bytes or line spans—and return no automatic citations. | +| Asynchronous processing | **Implemented with disclosed limits** | Analysis runs off the request path. Import, extraction of the initial archive/clone, and file-tree parsing remain synchronous; one in-process worker handles analysis jobs. | +| Incremental re-analysis and revision comparison | **Planned** | The full repository is analysed again; no snapshot-to-snapshot product workflow is available. | +| Change-impact or blast-radius analysis | **Implemented with disclosed limits** | Owner-scoped traversal over one sealed snapshot's resolved import and dependency edges. It does not compare revisions or calculate churn or trends. | +| Vulnerability and outdated-dependency scanning | **Planned** | Dependency responses report explicit `not_computed` states; Review keeps vulnerability scanning `not_assessed`. No clean bill of health or zero count is fabricated. | +| Grounded, cited free-form AI answers | **Planned** | Provider answers are intentionally uncited because providers do not receive source content or line numbers. | + +**Implemented with disclosed limits** means the workflow exists with an explicit coverage or trust boundary. **Planned** means it is roadmap work and current responses do not manufacture an answer. **Rejected** means the capability is intentionally outside the product contract. + ## One repository model, many consumers @@ -248,7 +250,7 @@ The prototype browser suite exercises defined Architecture, Engineering Review, ### Product limitations - **Pre-alpha, trusted-environment use.** PARTHA has not been operated or hardened as a public multi-tenant service. -- **Narrow semantic coverage.** Supported Python and TypeScript/JavaScript constructs receive the deepest extraction. Other languages primarily contribute file inventory. Role, module, layer, framework, and entry-point classifications can be heuristic. +- **Narrow semantic coverage.** The capability registry declares the Python and TypeScript/JavaScript constructs that receive the deepest extraction. Other languages primarily contribute file inventory. Role, module, layer, framework, and entry-point classifications can be heuristic. - **Narrow dependency coverage.** Only direct declarations in three manifest formats are extracted. Lockfiles, transitive dependencies, vulnerability scanning, and outdated-version scanning are not implemented. - **No repository evolution workflow.** Analysis is whole-repository; incremental analysis, revision comparison, and churn/trend analysis are unavailable. The sealed-snapshot impact query does not compare revisions or calculate historical change. - **Surface-dependent evidence.** A sealed snapshot does not make every product sentence line-cited. In particular, generated structural documentation and free-form AI have stricter evidence limits. diff --git a/apps/backend/app/extraction/manifests.py b/apps/backend/app/extraction/manifests.py index 8b6a8bb..af68e6b 100644 --- a/apps/backend/app/extraction/manifests.py +++ b/apps/backend/app/extraction/manifests.py @@ -26,8 +26,10 @@ logical_line_count, ) from app.intelligence import canonical +from app.extraction.support_matrix import supported_manifest_filenames +SUPPORTED_MANIFEST_FILENAMES = supported_manifest_filenames() _NPM_SECTIONS = ("dependencies", "devDependencies", "peerDependencies", "optionalDependencies") _NPM_DEPENDENCY_TYPES = { "dependencies": "production", @@ -84,7 +86,7 @@ def producer(self) -> str: return f"{self.name}@{self.version}" def supports(self, path: str) -> bool: - return posixpath.basename(path) in {"package.json", "pyproject.toml", "requirements.txt"} + return posixpath.basename(path) in SUPPORTED_MANIFEST_FILENAMES def extract(self, path: str, source: bytes) -> ExtractionResult: text, source_diagnostic = decode_source(path, source, producer=self.producer) diff --git a/apps/backend/app/extraction/support_matrix.py b/apps/backend/app/extraction/support_matrix.py index 7f44fc7..56af238 100644 --- a/apps/backend/app/extraction/support_matrix.py +++ b/apps/backend/app/extraction/support_matrix.py @@ -1,38 +1,866 @@ +"""The authoritative, typed capability registry for Repository Intelligence. + +The registry deliberately lives beside the production extractors. Benchmark +taxonomy is allowed to describe fixture variants, but it does not decide what +the product supports. ``SUPPORT_MATRIX`` remains as a compatibility view for +existing extractor tests and callers; its contents are derived from the +registry below. +""" + from __future__ import annotations from dataclasses import dataclass +from enum import StrEnum +from pathlib import Path +from typing import Iterable + + +REGISTRY_SCHEMA_VERSION = "ri-capability-registry.v1" +README_CAPABILITIES_START = "" +README_CAPABILITIES_END = "" + + +class SupportStatus(StrEnum): + SUPPORTED = "supported" + UNSUPPORTED = "unsupported" + LIMITED = "limited" + PARTIAL = "partial" + NOT_ASSESSED = "not_assessed" + + +class PublicStatus(StrEnum): + """Roadmap truth-boundary vocabulary used by public product claims.""" + + IMPLEMENTED = "Implemented" + IMPLEMENTED_WITH_DISCLOSED_LIMITS = "Implemented with disclosed limits" + PLANNED = "Planned" + REJECTED = "Rejected" + + +@dataclass(frozen=True) +class Capability: + """A production construct or explicitly disclosed production limitation.""" + + id: str + language: str + construct: str + status: SupportStatus + description: str + limitation: str + benchmark_ids: tuple[str, ...] = () + benchmark_disclosure: str | None = None + expected_diagnostic: str | None = None + + +@dataclass(frozen=True) +class PublicCapability: + """One row in README's public capability assessment.""" + + id: str + name: str + status: PublicStatus + statement: str + capability_ids: tuple[str, ...] + + +def _capability( + id: str, + language: str, + construct: str, + status: SupportStatus, + description: str, + limitation: str, + *benchmark_ids: str, + benchmark_disclosure: str | None = None, + expected_diagnostic: str | None = None, +) -> Capability: + return Capability( + id=id, + language=language, + construct=construct, + status=status, + description=description, + limitation=limitation, + benchmark_ids=benchmark_ids, + benchmark_disclosure=benchmark_disclosure, + expected_diagnostic=expected_diagnostic, + ) + + +# This is the only hand-maintained construct/status declaration. Keep entries +# sorted by stable id so registry serialization and all derived views are byte- +# stable. +CONSTRUCT_CAPABILITIES: tuple[Capability, ...] = ( + _capability( + "python.class", + "python", + "class", + SupportStatus.SUPPORTED, + "Class definitions.", + "Base-class semantics and metaclass behavior are not inferred.", + "py.class.def", + ), + _capability( + "python.decorator", + "python", + "decorator", + SupportStatus.SUPPORTED, + "Decorator observations and symbol properties.", + "Only syntax-visible decorator names are recorded; runtime decorator semantics are not evaluated.", + "py.decorator", + ), + _capability( + "python.dynamic-import", + "python", + "dynamic-import", + SupportStatus.UNSUPPORTED, + "Dynamic imports.", + "Dynamic import targets are not resolved.", + "py.dynamic_import", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "python.function", + "python", + "function", + SupportStatus.SUPPORTED, + "Function definitions, including async and nested definitions.", + "Signatures and runtime-generated functions are not modelled.", + "py.function.def", + "py.async_function.def", + "py.nested_function", + "py.duplicate_symbol", + ), + _capability( + "python.import", + "python", + "import", + SupportStatus.SUPPORTED, + "Python import and from-import observations.", + "Resolution is deliberately conservative and does not execute imports.", + "py.import", + "py.import_alias", + "py.from_import", + ), + _capability( + "python.metaclass", + "python", + "metaclass", + SupportStatus.UNSUPPORTED, + "Metaclass declarations.", + "Metaclass runtime behavior is not modelled.", + "py.metaclass", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "python.method", + "python", + "method", + SupportStatus.SUPPORTED, + "Methods defined inside classes.", + "Dynamic attributes and runtime method replacement are not modelled.", + "py.method.def", + ), + _capability( + "python.module", + "python", + "module", + SupportStatus.SUPPORTED, + "Directory-scoped Python module nodes.", + "Module identity is path-based.", + "py.module", + ), + _capability( + "python.monkeypatch", + "python", + "monkeypatch", + SupportStatus.UNSUPPORTED, + "Assignments that monkey-patch imported names.", + "Imported-name rebinding is reported but not interpreted as a stable relationship.", + "py.monkeypatch", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "python.reflection", + "python", + "reflection", + SupportStatus.UNSUPPORTED, + "Reflection calls such as getattr().", + "Reflection targets are not resolved.", + "py.reflection", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "python.route", + "python", + "route", + SupportStatus.SUPPORTED, + "Literal route paths on supported decorator forms.", + "The route contract covers syntax-visible literal paths only.", + "py.fastapi_route", + ), + _capability( + "python.star-import", + "python", + "star-import", + SupportStatus.UNSUPPORTED, + "Wildcard imports.", + "Star-import bindings are not expanded.", + "py.star_import", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "source.binary-file", + "source", + "binary-file", + SupportStatus.UNSUPPORTED, + "NUL-containing source files.", + "Binary files do not receive source line evidence.", + "src.binary_file", + expected_diagnostic="RI-SRC-BINARY", + ), + _capability( + "source.empty-file", + "source", + "empty-file", + SupportStatus.SUPPORTED, + "Empty text files with one logical line.", + "Empty files have whole-file evidence at logical line 1.", + "src.empty_file", + ), + _capability( + "source.file", + "source", + "file", + SupportStatus.SUPPORTED, + "Repository inventory file nodes.", + "Inventory is not semantic extraction for unsupported languages.", + "src.file", + ), + _capability( + "source.large-file", + "source", + "large-file", + SupportStatus.UNSUPPORTED, + "Files above the configured source budget.", + "Oversized source is skipped rather than retained for extraction.", + "src.large_file", + expected_diagnostic="RI-LIMIT-SKIP", + ), + _capability( + "source.malformed-source", + "source", + "malformed-source", + SupportStatus.UNSUPPORTED, + "Undecodable or malformed source.", + "Malformed input produces a diagnostic and no fabricated facts.", + "py.syntax_error", + "ts.syntax_error", + "src.malformed_source", + expected_diagnostic="RI-SRC-MALFORMED", + ), + _capability( + "source.path-escape", + "source", + "path-escape", + SupportStatus.UNSUPPORTED, + "Paths that are absolute or escape the repository root.", + "Unsafe paths are rejected before extraction.", + "src.path_escape", + expected_diagnostic="RI-SEC-PATH-ESCAPE", + ), + _capability( + "source.repository", + "source", + "repository", + SupportStatus.SUPPORTED, + "The repository root inventory node.", + "The node identifies the analyzed revision; it is not a source-language claim.", + "src.repository", + ), + _capability( + "source.trailing-newline", + "source", + "trailing-newline", + SupportStatus.SUPPORTED, + "Logical line accounting for trailing newlines.", + "Line accounting follows the ri.v1 empty/trailing-newline rules.", + "src.trailing_newline", + ), + _capability( + "typescript.ambient-module", + "typescript", + "ambient-module", + SupportStatus.UNSUPPORTED, + "Ambient module declarations.", + "Ambient module semantics are not extracted.", + "ts.ambient_module", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "typescript.class", + "typescript", + "class", + SupportStatus.SUPPORTED, + "TypeScript class declarations.", + "Type information and runtime inheritance are not inferred beyond emitted syntax facts.", + "ts.class", + ), + _capability( + "typescript.commonjs-require", + "typescript", + "commonjs-require", + SupportStatus.UNSUPPORTED, + "CommonJS require() calls.", + "CommonJS binding semantics are not resolved by the TypeScript extractor.", + "ts.commonjs_require", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "typescript.const", + "typescript", + "const", + SupportStatus.SUPPORTED, + "Top-level const bindings.", + "Only the supported declaration shape is emitted.", + "ts.const", + ), + _capability( + "typescript.decorator", + "typescript", + "decorator", + SupportStatus.UNSUPPORTED, + "TypeScript decorators.", + "Decorator runtime semantics are not extracted.", + "ts.decorator", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "typescript.dynamic-import", + "typescript", + "dynamic-import", + SupportStatus.UNSUPPORTED, + "Dynamic import expressions.", + "Dynamic import targets are not resolved.", + "ts.dynamic_import", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "typescript.enum", + "typescript", + "enum", + SupportStatus.SUPPORTED, + "Enum declarations.", + "Enum runtime expansion is not inferred.", + "ts.enum", + ), + _capability( + "typescript.export", + "typescript", + "export", + SupportStatus.SUPPORTED, + "Export observations.", + "Exports are recorded syntactically without module execution.", + "ts.export", + "ts.reexport", + ), + _capability( + "typescript.file", + "typescript", + "file", + SupportStatus.SUPPORTED, + "TypeScript file nodes.", + "File nodes carry inventory and language facts; semantic extraction remains construct-specific.", + "ts.module", + ), + _capability( + "typescript.function", + "typescript", + "function", + SupportStatus.SUPPORTED, + "Function declarations, including async functions.", + "Type inference and runtime-generated functions are not modelled.", + "ts.function", + "ts.async_function", + ), + _capability( + "typescript.import", + "typescript", + "import", + SupportStatus.SUPPORTED, + "TypeScript import observations and bindings.", + "Resolution is syntax- and snapshot-fact-based; source is not executed.", + "ts.import", + "ts.import_alias", + ), + _capability( + "typescript.interface", + "typescript", + "interface", + SupportStatus.SUPPORTED, + "Interface declarations.", + "Structural type checking is not performed.", + "ts.interface", + ), + _capability( + "typescript.method", + "typescript", + "method", + SupportStatus.SUPPORTED, + "Class methods.", + "Dynamic dispatch and runtime replacement are not modelled.", + "ts.method", + ), + _capability( + "typescript.module", + "typescript", + "module", + SupportStatus.SUPPORTED, + "Directory-scoped TypeScript module nodes.", + "Module identity is path-based.", + "ts.directory_module", + ), + _capability( + "typescript.namespace", + "typescript", + "namespace", + SupportStatus.UNSUPPORTED, + "Namespace/module declarations.", + "Namespace runtime and merge semantics are not extracted.", + "ts.namespace", + expected_diagnostic="RI-EXT-UNSUPPORTED", + ), + _capability( + "typescript.route", + "typescript", + "route", + SupportStatus.SUPPORTED, + "Literal routes in supported react-router forms.", + "Dynamic route paths and unrelated path-shaped objects are not treated as routes.", + "ts.route", + ), + _capability( + "typescript.type", + "typescript", + "type", + SupportStatus.SUPPORTED, + "Type alias declarations.", + "Type evaluation is not performed.", + "ts.type", + ), +) + + +MANIFEST_CAPABILITIES: tuple[Capability, ...] = ( + _capability( + "dependency.package-json", + "source", + "manifest:package.json", + SupportStatus.SUPPORTED, + "Direct npm declarations in package.json.", + "Only direct declarations are extracted; lockfiles, transitive resolution, vulnerability scanning, and outdated-version scanning are not implemented.", + benchmark_disclosure="No benchmark fixture currently exercises dependency-manifest extraction; production integration tests cover this extractor.", + ), + _capability( + "dependency.pyproject", + "source", + "manifest:pyproject.toml", + SupportStatus.SUPPORTED, + "Direct PyPI declarations in pyproject.toml.", + "Only supported project.dependencies declarations are extracted; transitive resolution and scanning are not implemented.", + benchmark_disclosure="No benchmark fixture currently exercises dependency-manifest extraction; production integration tests cover this extractor.", + ), + _capability( + "dependency.requirements", + "source", + "manifest:requirements.txt", + SupportStatus.SUPPORTED, + "Direct PyPI declarations in requirements.txt.", + "Only direct requirement lines are extracted; transitive resolution and scanning are not implemented.", + benchmark_disclosure="No benchmark fixture currently exercises dependency-manifest extraction; production integration tests cover this extractor.", + ), +) + + +def _product_capability( + id: str, + status: SupportStatus, + description: str, + limitation: str, +) -> Capability: + return _capability( + f"product.{id}", + "product", + id, + status, + description, + limitation, + benchmark_disclosure=( + "This product-level claim is verified by its focused integration or acceptance " + "tests rather than the Repository Intelligence golden benchmark." + ), + ) + + +PRODUCT_CAPABILITIES: tuple[Capability, ...] = ( + _product_capability( + "ai-provider", + SupportStatus.LIMITED, + "AI provider integration.", + "Provider answers receive bounded structural context and do not automatically carry source citations.", + ), + _product_capability( + "analysis-lifecycle", + SupportStatus.SUPPORTED, + "Database-backed analysis lifecycle.", + "Execution remains bounded by the configured worker and retry policy.", + ), + _product_capability( + "architecture-authentication", + SupportStatus.LIMITED, + "Architecture and authentication explanation.", + "Classification and authentication coverage are deliberately heuristic and pattern-bounded.", + ), + _product_capability( + "archive-import", + SupportStatus.SUPPORTED, + "Archive upload and public GitHub import.", + "Private GitHub cloning and non-GitHub repository hosts are not supported.", + ), + _product_capability( + "async-processing", + SupportStatus.PARTIAL, + "Asynchronous analysis processing.", + "Import, initial extraction, and file-tree parsing remain synchronous.", + ), + _product_capability( + "authentication-isolation", + SupportStatus.SUPPORTED, + "Authentication and owner isolation.", + "Protected resources deliberately conceal non-owner existence with 404 responses.", + ), + _product_capability( + "change-impact", + SupportStatus.LIMITED, + "Change-impact traversal.", + "Traversal covers one sealed snapshot and does not compare revisions.", + ), + _product_capability( + "dependency-graph", + SupportStatus.LIMITED, + "Direct dependency graph.", + "Lockfiles, transitive resolution, vulnerability scanning, and outdated-version scanning are not implemented.", + ), + _product_capability( + "dependency-scanning", + SupportStatus.NOT_ASSESSED, + "Vulnerability and outdated-dependency scanning.", + "This capability is planned; current responses explicitly publish not-computed or not-assessed states.", + ), + _product_capability( + "documentation-export", + SupportStatus.LIMITED, + "Documentation and report export.", + "Documentation uses current-revision structural facts.", + ), + _product_capability( + "engineering-review", + SupportStatus.LIMITED, + "Evidence-addressed engineering review.", + "No overall score, grade, health percentage, vulnerability result, or generated roadmap is produced.", + ), + _product_capability( + "grounded-ai", + SupportStatus.NOT_ASSESSED, + "Grounded, cited free-form AI answers.", + "This capability is planned; current provider answers intentionally remain uncited.", + ), + _product_capability( + "repository-explorer", + SupportStatus.SUPPORTED, + "Owner-scoped repository explorer.", + "Preview is bounded and binary-aware.", + ), + _product_capability( + "repository-insights", + SupportStatus.LIMITED, + "Repository insights.", + "Insights describe one snapshot and make no change-over-time claims.", + ), + _product_capability( + "repository-intelligence", + SupportStatus.LIMITED, + "Revision-addressed Repository Intelligence snapshots.", + "Semantic extraction is strongest for supported Python and TypeScript/JavaScript constructs.", + ), + _product_capability( + "revision-comparison", + SupportStatus.NOT_ASSESSED, + "Incremental re-analysis and revision comparison.", + "This capability is planned; the current workflow re-analyses the full repository.", + ), +) + + +_SUPPORTED_MANIFEST_FILENAMES = tuple( + sorted( + item.construct.removeprefix("manifest:") + for item in MANIFEST_CAPABILITIES + if item.status == SupportStatus.SUPPORTED + ) +) + + +def _code_list(items: tuple[str, ...]) -> str: + rendered = [f"`{item}`" for item in items] + if len(rendered) == 1: + return rendered[0] + return f"{', '.join(rendered[:-1])}, and {rendered[-1]}" + + +PUBLIC_CAPABILITIES: tuple[PublicCapability, ...] = ( + PublicCapability( + "archive-import", + "Archive upload and public GitHub import", + PublicStatus.IMPLEMENTED, + "ZIP/TAR-family archives and shallow public GitHub HTTPS clones; size and path-safety limits apply. Private GitHub cloning and other repository hosts are not supported.", + ("product.archive-import",), + ), + PublicCapability( + "repository-explorer", + "Repository explorer", + PublicStatus.IMPLEMENTED, + "Owner-scoped file tree plus bounded text/image preview, binary detection, and truncation.", + ("product.repository-explorer",), + ), + PublicCapability( + "authentication-isolation", + "Authentication and owner isolation", + PublicStatus.IMPLEMENTED, + "Email/password, Argon2, short-lived access tokens, rotating refresh tokens with reuse detection. Protected resources are owner-scoped; non-owner access returns 404.", + ("product.authentication-isolation",), + ), + PublicCapability( + "analysis-lifecycle", + "Analysis lifecycle", + PublicStatus.IMPLEMENTED, + "Database-backed, cancellable job with progress, bounded retry, lease renewal, and stale-worker recovery.", + ("product.analysis-lifecycle",), + ), + PublicCapability( + "repository-intelligence", + "Repository Intelligence", + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS, + "Immutable, revision-addressed `ri.v1` snapshots with normalized facts, evidence, query APIs, and a total canonical graph hash. Semantic extraction is strongest for supported Python and TypeScript/JavaScript constructs.", + ("product.repository-intelligence",), + ), + PublicCapability( + "architecture-authentication", + "Architecture and authentication explanation", + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS, + "Interactive snapshot-backed graph. Module/layer classification is heuristic. The cited authentication subgraph covers supported Python/FastAPI patterns only.", + ("product.architecture-authentication",), + ), + PublicCapability( + "dependency-graph", + "Dependency Graph", + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS, + f"Direct declarations from {_code_list(_SUPPORTED_MANIFEST_FILENAMES)}, including repeated workspace declarations and exact manifest spans. No lockfiles or transitive resolution.", + ("product.dependency-graph", *(item.id for item in MANIFEST_CAPABILITIES)), + ), + PublicCapability( + "engineering-review", + "Engineering Review", + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS, + "`engineering-review.v2`; evidence-addressed findings and explicit category states. No overall score, grade, health percentage, vulnerability result, or generated roadmap.", + ("product.engineering-review",), + ), + PublicCapability( + "repository-insights", + "Repository Insights", + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS, + "`repository-insights.v1`; defined counts, ratios, diagnostics, language breakdowns, and extraction coverage from one snapshot. No change-over-time claims.", + ("product.repository-insights",), + ), + PublicCapability( + "documentation-export", + "Documentation and report export", + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS, + "Documentation uses current-revision structural facts. Review, Documentation, Architecture, and Dependencies export through one JSON/Markdown/HTML/PDF pipeline.", + ("product.documentation-export",), + ), + PublicCapability( + "ai-provider", + "AI provider integration", + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS, + "Per-user configuration for supported providers, encrypted API keys, and constrained outbound destinations. Free-form answers receive structural facts and observed paths\u2014not source bytes or line spans\u2014and return no automatic citations.", + ("product.ai-provider",), + ), + PublicCapability( + "async-processing", + "Asynchronous processing", + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS, + "Analysis runs off the request path. Import, extraction of the initial archive/clone, and file-tree parsing remain synchronous; one in-process worker handles analysis jobs.", + ("product.async-processing",), + ), + PublicCapability( + "revision-comparison", + "Incremental re-analysis and revision comparison", + PublicStatus.PLANNED, + "The full repository is analysed again; no snapshot-to-snapshot product workflow is available.", + ("product.revision-comparison",), + ), + PublicCapability( + "change-impact", + "Change-impact or blast-radius analysis", + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS, + "Owner-scoped traversal over one sealed snapshot's resolved import and dependency edges. It does not compare revisions or calculate churn or trends.", + ("product.change-impact",), + ), + PublicCapability( + "dependency-scanning", + "Vulnerability and outdated-dependency scanning", + PublicStatus.PLANNED, + "Dependency responses report explicit `not_computed` states; Review keeps vulnerability scanning `not_assessed`. No clean bill of health or zero count is fabricated.", + ("product.dependency-scanning",), + ), + PublicCapability( + "grounded-ai", + "Grounded, cited free-form AI answers", + PublicStatus.PLANNED, + "Provider answers are intentionally uncited because providers do not receive source content or line numbers.", + ("product.grounded-ai",), + ), +) + + +CAPABILITY_REGISTRY: tuple[Capability, ...] = tuple( + sorted( + (*CONSTRUCT_CAPABILITIES, *MANIFEST_CAPABILITIES, *PRODUCT_CAPABILITIES), + key=lambda item: item.id, + ) +) +CAPABILITIES_BY_ID = {item.id: item for item in CAPABILITY_REGISTRY} @dataclass(frozen=True) class LanguageSupport: + """Compatibility projection consumed by existing extractor tests.""" + supported: tuple[str, ...] unsupported: tuple[str, ...] +def validate_registry( + capabilities: Iterable[Capability] = CAPABILITY_REGISTRY, + public_capabilities: Iterable[PublicCapability] = PUBLIC_CAPABILITIES, +) -> None: + entries = tuple(capabilities) + ids = [item.id for item in entries] + if ids != sorted(ids): + raise ValueError("capability registry entries must be ordered by stable id") + if len(ids) != len(set(ids)): + raise ValueError("capability registry contains duplicate capability ids") + for item in entries: + if not item.id or not item.language or not item.construct: + raise ValueError(f"capability {item.id!r} is missing a stable identity") + if not isinstance(item.status, SupportStatus): + raise ValueError(f"capability {item.id!r} has an unsupported status") + if not item.description or not item.limitation: + raise ValueError(f"capability {item.id!r} must disclose description and limitation") + if item.status == SupportStatus.UNSUPPORTED and not item.expected_diagnostic: + raise ValueError(f"unsupported capability {item.id!r} must name its diagnostic") + if item.status != SupportStatus.UNSUPPORTED and item.expected_diagnostic: + raise ValueError(f"supported capability {item.id!r} cannot require an unsupported diagnostic") + if not item.benchmark_ids and not item.benchmark_disclosure: + raise ValueError(f"capability {item.id!r} needs benchmark coverage or an explicit disclosure") + if len(item.benchmark_ids) != len(set(item.benchmark_ids)): + raise ValueError(f"capability {item.id!r} contains duplicate benchmark ids") + + capabilities_by_id = {item.id: item for item in entries} + allowed_statuses = { + PublicStatus.IMPLEMENTED: {SupportStatus.SUPPORTED}, + PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS: { + SupportStatus.SUPPORTED, + SupportStatus.LIMITED, + SupportStatus.PARTIAL, + }, + PublicStatus.PLANNED: {SupportStatus.NOT_ASSESSED}, + PublicStatus.REJECTED: {SupportStatus.UNSUPPORTED}, + } + for public in public_capabilities: + if not isinstance(public.status, PublicStatus): + raise ValueError(f"public capability {public.id!r} has an unsupported status") + if not public.capability_ids: + raise ValueError(f"public capability {public.id!r} must resolve to registry capabilities") + resolved = [] + for capability_id in public.capability_ids: + capability = capabilities_by_id.get(capability_id) + if capability is None: + raise ValueError(f"public capability {public.id!r} references unknown capability {capability_id!r}") + resolved.append(capability) + if any(item.status not in allowed_statuses[public.status] for item in resolved): + raise ValueError(f"public capability {public.id!r} status is inconsistent with its registry capabilities") + if public.status == PublicStatus.IMPLEMENTED_WITH_DISCLOSED_LIMITS and not any( + item.status in {SupportStatus.LIMITED, SupportStatus.PARTIAL} for item in resolved + ): + raise ValueError(f"public capability {public.id!r} must resolve to a disclosed limited capability") + + +# Deliberate fail-fast: registry drift is an application-startup error because +# extractors and public claims must never run from an invalid truth contract. +validate_registry() + +_MATRIX_CAPABILITIES = tuple( + item + for item in CAPABILITY_REGISTRY + if item.language in {"python", "source", "typescript"} and not item.construct.startswith("manifest:") +) SUPPORT_MATRIX: dict[str, LanguageSupport] = { - "python": LanguageSupport( - supported=("module", "import", "function", "class", "method", "decorator", "route"), - unsupported=("star-import", "dynamic-import", "reflection", "monkeypatch", "metaclass"), - ), - "typescript": LanguageSupport( - supported=( - "file", - "module", - "import", - "export", - "function", - "class", - "method", - "interface", - "type", - "enum", - "const", - "route", + language: LanguageSupport( + supported=tuple( + item.construct + for item in _MATRIX_CAPABILITIES + if item.language == language and item.status == SupportStatus.SUPPORTED ), - unsupported=("dynamic-import", "decorator", "namespace", "commonjs-require", "ambient-module"), - ), - "source": LanguageSupport( - supported=("repository", "file", "empty-file", "trailing-newline"), - unsupported=("binary-file", "malformed-source", "large-file", "path-escape"), - ), + unsupported=tuple( + item.construct + for item in _MATRIX_CAPABILITIES + if item.language == language and item.status == SupportStatus.UNSUPPORTED + ), + ) + for language in sorted({item.language for item in _MATRIX_CAPABILITIES}) } + + +def supported_manifest_filenames() -> tuple[str, ...]: + return _SUPPORTED_MANIFEST_FILENAMES + + +def render_readme_capabilities() -> str: + lines = [ + README_CAPABILITIES_START, + "| Capability | Status | Current boundary |", + "| --- | --- | --- |", + ] + lines.extend(f"| {item.name} | **{item.status}** | {item.statement} |" for item in PUBLIC_CAPABILITIES) + lines.extend( + [ + "", + "**Implemented with disclosed limits** means the workflow exists with an explicit coverage or trust boundary. **Planned** means it is roadmap work and current responses do not manufacture an answer. **Rejected** means the capability is intentionally outside the product contract.", + README_CAPABILITIES_END, + ] + ) + return "\n".join(lines) + + +def check_readme_capabilities(readme: Path) -> None: + content = readme.read_text(encoding="utf-8") + start = content.find(README_CAPABILITIES_START) + end = content.find(README_CAPABILITIES_END) + if start < 0 or end < start: + raise ValueError("README is missing the generated capability registry markers") + actual = content[start : end + len(README_CAPABILITIES_END)] + expected = render_readme_capabilities() + if actual != expected: + raise ValueError("README capability registry block is stale; run the reviewed registry renderer") diff --git a/apps/backend/app/graph/dependency_graph.py b/apps/backend/app/graph/dependency_graph.py index 2140aa9..2ec6d8b 100644 --- a/apps/backend/app/graph/dependency_graph.py +++ b/apps/backend/app/graph/dependency_graph.py @@ -16,6 +16,7 @@ from app.analysis.manifest import build_manifest, manifest_digest from app.intelligence.query_service import SnapshotQueryService +from app.extraction.support_matrix import supported_manifest_filenames from app.models.repository import RepositoryRecord from app.models.snapshot import RiDiagnostic, RiEdge, RiEvidence, RiNode from app.schemas.dependencies import ( @@ -31,7 +32,7 @@ #: Manifest filenames ``DependencyManifestExtractor`` supports (app/extraction/manifests.py). #: A diagnostic is "relevant to dependency extraction" only when it names one of these paths — #: the same scoping the legacy engine got for free by only ever iterating manifest-supporting files. -_MANIFEST_FILENAMES = frozenset({"package.json", "pyproject.toml", "requirements.txt"}) +_MANIFEST_FILENAMES = frozenset(supported_manifest_filenames()) #: Codes the manifest extractor and the pipeline's size-budget gate can emit for a manifest file. _RELEVANT_DIAGNOSTIC_CODES = frozenset({"RI-SRC-MALFORMED", "RI-LIMIT-SKIP"}) diff --git a/apps/backend/tests/benchmark/README.md b/apps/backend/tests/benchmark/README.md index bff3ec1..6df3cb4 100644 --- a/apps/backend/tests/benchmark/README.md +++ b/apps/backend/tests/benchmark/README.md @@ -17,7 +17,7 @@ into the actual side. | Path | What it is | | --- | --- | | [`fixtures/{minimal,realistic,adversarial}/`](fixtures) | The versioned golden corpus: synthetic source + a `manifest.json` of expected facts per fixture. | -| [`config/benchmark_support_matrix.json`](config/benchmark_support_matrix.json) | The benchmark construct taxonomy and validated mapping to every production support-matrix entry. | +| [`config/benchmark_support_matrix.json`](config/benchmark_support_matrix.json) | Benchmark-only construct taxonomy mapped to every production capability id; status and limitations come from the production registry. | | [`config/thresholds.json`](config/thresholds.json) | The versioned, exact-fraction acceptance thresholds. | | [`facts.py`](facts.py) / [`scorer.py`](scorer.py) | The fact model and precision/recall scorer. | | [`loader.py`](loader.py) / [`schema.py`](schema.py) | Strict manifest loading and validation. | @@ -39,7 +39,11 @@ python tests/benchmark/run.py --report-dir /tmp/ri-benchmark cat /tmp/ri-benchmark/benchmark.md ``` -The runner exits non-zero when any enforced gate fails. **Reports are generated, +The runner exits non-zero when any enforced gate fails. This includes an exact +golden regression comparison: a real-extractor fact/evidence change cannot pass +only because aggregate precision and recall remain above threshold. An +intentional baseline change requires a reviewed hand-authored fixture or +expectation diff. **Reports are generated, never committed** (`.gitignore` excludes them; CI writes them to `$RUNNER_TEMP`). ## Fixture schema and versioning @@ -95,7 +99,7 @@ getting maintainer agreement. ## How unsupported constructs are scored -A construct outside the support matrix must produce **no fact** plus a specific +A construct outside the production capability registry must produce **no fact** plus a specific diagnostic (e.g. `RI-EXT-UNSUPPORTED`, `RI-SRC-MALFORMED`). In scoring: - an invented fact where none is expected is a **false positive**; @@ -106,10 +110,10 @@ diagnostic (e.g. `RI-EXT-UNSUPPORTED`, `RI-SRC-MALFORMED`). In scoring: ## Adding or reviewing a fixture 1. Write small, original synthetic source (no third-party/copyrighted code). -2. Derive the expected facts **by hand** from the source and the support matrix — +2. Derive the expected facts **by hand** from the source and the production capability registry — count the one-based line spans yourself. Do **not** run any extractor and copy its output. -3. Add the manifest; tag each fact with the support-matrix construct(s) it covers. +3. Add the manifest; tag each fact with the benchmark construct id(s) it covers. 4. `python -m pytest tests/benchmark` — the loader validates structure and provenance; the runner validates parity, diagnostics, and determinism. @@ -134,7 +138,7 @@ page, and fails the job when the benchmark is below threshold. - every citation emitted by the real extractors is valid; - repeated real extraction, including reversed insertion order, seals to the same product canonical graph hash; -- support-matrix drift, duplicate/cross-fixture facts, invalid citations, bad +- capability-registry drift, duplicate/cross-fixture facts, invalid citations, bad precision/recall, and nondeterminism fail the build. Durable product analysis uses the same extractors, support matrices, resolver, diff --git a/apps/backend/tests/benchmark/config/benchmark_support_matrix.json b/apps/backend/tests/benchmark/config/benchmark_support_matrix.json index b6655dd..bf6ebc7 100644 --- a/apps/backend/tests/benchmark/config/benchmark_support_matrix.json +++ b/apps/backend/tests/benchmark/config/benchmark_support_matrix.json @@ -1,102 +1,102 @@ { - "schemaVersion": "ri-benchmark-support-matrix.v1", - "note": "Benchmark construct names are explicitly mapped to the authoritative production extractor/source-policy matrices. Several benchmark ids intentionally map to one production construct when the corpus tests a semantic variant such as async, aliasing, nesting, or duplicate identity.", + "schemaVersion": "ri-benchmark-support-matrix.v2", + "note": "Benchmark ids are fixture taxonomy only. Each id maps to exactly one authoritative production capability id; support status, language, limitation, and required diagnostic come from app/extraction/support_matrix.py.", "constructs": { - "py.module": {"language": "python", "supported": true, "description": "A directory-scoped Python module node."}, - "py.function.def": {"language": "python", "supported": true, "description": "A top-level function definition."}, - "py.async_function.def": {"language": "python", "supported": true, "description": "A top-level async function definition."}, - "py.class.def": {"language": "python", "supported": true, "description": "A class definition."}, - "py.method.def": {"language": "python", "supported": true, "description": "A method defined inside a class."}, - "py.nested_function": {"language": "python", "supported": true, "description": "A function nested inside another function."}, - "py.duplicate_symbol": {"language": "python", "supported": true, "description": "A redefined name resolved with an ordinal discriminator."}, - "py.import": {"language": "python", "supported": true, "description": "A plain import statement."}, - "py.import_alias": {"language": "python", "supported": true, "description": "An aliased import statement."}, - "py.from_import": {"language": "python", "supported": true, "description": "A from-import statement."}, - "py.decorator": {"language": "python", "supported": true, "description": "A decorator observation and symbol property."}, - "py.fastapi_route": {"language": "python", "supported": true, "description": "A FastAPI decorator route observation."}, - "py.dynamic_import": {"language": "python", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "Dynamic import."}, - "py.monkeypatch": {"language": "python", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "Attribute rebinding on an imported name."}, - "py.reflection": {"language": "python", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "Reflection calls."}, - "py.star_import": {"language": "python", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "Wildcard import."}, - "py.metaclass": {"language": "python", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "Metaclass declaration."}, - "py.syntax_error": {"language": "python", "supported": false, "expectedDiagnostic": "RI-SRC-MALFORMED", "description": "Malformed Python source."}, - "ts.module": {"language": "typescript", "supported": true, "description": "A TypeScript file node."}, - "ts.directory_module": {"language": "typescript", "supported": true, "description": "A directory-scoped TypeScript module node."}, - "ts.function": {"language": "typescript", "supported": true, "description": "A function declaration."}, - "ts.async_function": {"language": "typescript", "supported": true, "description": "An async function declaration."}, - "ts.class": {"language": "typescript", "supported": true, "description": "A class declaration."}, - "ts.method": {"language": "typescript", "supported": true, "description": "A class method."}, - "ts.interface": {"language": "typescript", "supported": true, "description": "An interface declaration."}, - "ts.type": {"language": "typescript", "supported": true, "description": "A type alias declaration."}, - "ts.enum": {"language": "typescript", "supported": true, "description": "An enum declaration."}, - "ts.const": {"language": "typescript", "supported": true, "description": "A top-level const binding."}, - "ts.import": {"language": "typescript", "supported": true, "description": "An import statement."}, - "ts.import_alias": {"language": "typescript", "supported": true, "description": "An aliased import statement."}, - "ts.export": {"language": "typescript", "supported": true, "description": "An exported symbol property."}, - "ts.reexport": {"language": "typescript", "supported": true, "description": "A re-export import observation."}, - "ts.route": {"language": "typescript", "supported": true, "description": "A react-router route observation."}, - "ts.dynamic_import": {"language": "typescript", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "Dynamic import expression."}, - "ts.decorator": {"language": "typescript", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "TypeScript decorator."}, - "ts.namespace": {"language": "typescript", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "Namespace declaration."}, - "ts.commonjs_require": {"language": "typescript", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "CommonJS require call."}, - "ts.ambient_module": {"language": "typescript", "supported": false, "expectedDiagnostic": "RI-EXT-UNSUPPORTED", "description": "Ambient module declaration."}, - "ts.syntax_error": {"language": "typescript", "supported": false, "expectedDiagnostic": "RI-SRC-MALFORMED", "description": "Malformed TypeScript source."}, - "src.repository": {"language": "mixed", "supported": true, "description": "The repository root inventory node."}, - "src.file": {"language": "mixed", "supported": true, "description": "A repository inventory file node."}, - "src.empty_file": {"language": "mixed", "supported": true, "description": "A zero-byte text file with one logical line."}, - "src.trailing_newline": {"language": "mixed", "supported": true, "description": "A trailing newline contributes the final logical line."}, - "src.binary_file": {"language": "mixed", "supported": false, "expectedDiagnostic": "RI-SRC-BINARY", "description": "A NUL-containing source file."}, - "src.malformed_source": {"language": "mixed", "supported": false, "expectedDiagnostic": "RI-SRC-MALFORMED", "description": "Undecodable source bytes."}, - "src.large_file": {"language": "mixed", "supported": false, "expectedDiagnostic": "RI-LIMIT-SKIP", "description": "A source above the configured byte budget."}, - "src.path_escape": {"language": "mixed", "supported": false, "expectedDiagnostic": "RI-SEC-PATH-ESCAPE", "description": "A source path escaping the repository root."} + "py.module": {"description": "A directory-scoped Python module node."}, + "py.function.def": {"description": "A top-level function definition."}, + "py.async_function.def": {"description": "A top-level async function definition."}, + "py.class.def": {"description": "A class definition."}, + "py.method.def": {"description": "A method defined inside a class."}, + "py.nested_function": {"description": "A function nested inside another function."}, + "py.duplicate_symbol": {"description": "A redefined name resolved with an ordinal discriminator."}, + "py.import": {"description": "A plain import statement."}, + "py.import_alias": {"description": "An aliased import statement."}, + "py.from_import": {"description": "A from-import statement."}, + "py.decorator": {"description": "A decorator observation and symbol property."}, + "py.fastapi_route": {"description": "A FastAPI decorator route observation."}, + "py.dynamic_import": {"description": "Dynamic import."}, + "py.monkeypatch": {"description": "Attribute rebinding on an imported name."}, + "py.reflection": {"description": "Reflection calls."}, + "py.star_import": {"description": "Wildcard import."}, + "py.metaclass": {"description": "Metaclass declaration."}, + "py.syntax_error": {"description": "Malformed Python source."}, + "ts.module": {"description": "A TypeScript file node."}, + "ts.directory_module": {"description": "A directory-scoped TypeScript module node."}, + "ts.function": {"description": "A function declaration."}, + "ts.async_function": {"description": "An async function declaration."}, + "ts.class": {"description": "A class declaration."}, + "ts.method": {"description": "A class method."}, + "ts.interface": {"description": "An interface declaration."}, + "ts.type": {"description": "A type alias declaration."}, + "ts.enum": {"description": "An enum declaration."}, + "ts.const": {"description": "A top-level const binding."}, + "ts.import": {"description": "An import statement."}, + "ts.import_alias": {"description": "An aliased import statement."}, + "ts.export": {"description": "An exported symbol property."}, + "ts.reexport": {"description": "A re-export import observation."}, + "ts.route": {"description": "A react-router route observation."}, + "ts.dynamic_import": {"description": "Dynamic import expression."}, + "ts.decorator": {"description": "TypeScript decorator."}, + "ts.namespace": {"description": "Namespace declaration."}, + "ts.commonjs_require": {"description": "CommonJS require call."}, + "ts.ambient_module": {"description": "Ambient module declaration."}, + "ts.syntax_error": {"description": "Malformed TypeScript source."}, + "src.repository": {"description": "The repository root inventory node."}, + "src.file": {"description": "A repository inventory file node."}, + "src.empty_file": {"description": "A zero-byte text file with one logical line."}, + "src.trailing_newline": {"description": "A trailing newline contributes the final logical line."}, + "src.binary_file": {"description": "A NUL-containing source file."}, + "src.malformed_source": {"description": "Undecodable source bytes."}, + "src.large_file": {"description": "A source above the configured byte budget."}, + "src.path_escape": {"description": "A source path escaping the repository root."} }, "productionMappings": { - "py.module": {"language": "python", "construct": "module"}, - "py.function.def": {"language": "python", "construct": "function"}, - "py.async_function.def": {"language": "python", "construct": "function"}, - "py.class.def": {"language": "python", "construct": "class"}, - "py.method.def": {"language": "python", "construct": "method"}, - "py.nested_function": {"language": "python", "construct": "function"}, - "py.duplicate_symbol": {"language": "python", "construct": "function"}, - "py.import": {"language": "python", "construct": "import"}, - "py.import_alias": {"language": "python", "construct": "import"}, - "py.from_import": {"language": "python", "construct": "import"}, - "py.decorator": {"language": "python", "construct": "decorator"}, - "py.fastapi_route": {"language": "python", "construct": "route"}, - "py.dynamic_import": {"language": "python", "construct": "dynamic-import"}, - "py.monkeypatch": {"language": "python", "construct": "monkeypatch"}, - "py.reflection": {"language": "python", "construct": "reflection"}, - "py.star_import": {"language": "python", "construct": "star-import"}, - "py.metaclass": {"language": "python", "construct": "metaclass"}, - "py.syntax_error": {"language": "source", "construct": "malformed-source"}, - "ts.module": {"language": "typescript", "construct": "file"}, - "ts.directory_module": {"language": "typescript", "construct": "module"}, - "ts.function": {"language": "typescript", "construct": "function"}, - "ts.async_function": {"language": "typescript", "construct": "function"}, - "ts.class": {"language": "typescript", "construct": "class"}, - "ts.method": {"language": "typescript", "construct": "method"}, - "ts.interface": {"language": "typescript", "construct": "interface"}, - "ts.type": {"language": "typescript", "construct": "type"}, - "ts.enum": {"language": "typescript", "construct": "enum"}, - "ts.const": {"language": "typescript", "construct": "const"}, - "ts.import": {"language": "typescript", "construct": "import"}, - "ts.import_alias": {"language": "typescript", "construct": "import"}, - "ts.export": {"language": "typescript", "construct": "export"}, - "ts.reexport": {"language": "typescript", "construct": "import"}, - "ts.route": {"language": "typescript", "construct": "route"}, - "ts.dynamic_import": {"language": "typescript", "construct": "dynamic-import"}, - "ts.decorator": {"language": "typescript", "construct": "decorator"}, - "ts.namespace": {"language": "typescript", "construct": "namespace"}, - "ts.commonjs_require": {"language": "typescript", "construct": "commonjs-require"}, - "ts.ambient_module": {"language": "typescript", "construct": "ambient-module"}, - "ts.syntax_error": {"language": "source", "construct": "malformed-source"}, - "src.repository": {"language": "source", "construct": "repository"}, - "src.file": {"language": "source", "construct": "file"}, - "src.empty_file": {"language": "source", "construct": "empty-file"}, - "src.trailing_newline": {"language": "source", "construct": "trailing-newline"}, - "src.binary_file": {"language": "source", "construct": "binary-file"}, - "src.malformed_source": {"language": "source", "construct": "malformed-source"}, - "src.large_file": {"language": "source", "construct": "large-file"}, - "src.path_escape": {"language": "source", "construct": "path-escape"} + "py.module": "python.module", + "py.function.def": "python.function", + "py.async_function.def": "python.function", + "py.nested_function": "python.function", + "py.duplicate_symbol": "python.function", + "py.class.def": "python.class", + "py.method.def": "python.method", + "py.import": "python.import", + "py.import_alias": "python.import", + "py.from_import": "python.import", + "py.decorator": "python.decorator", + "py.fastapi_route": "python.route", + "py.dynamic_import": "python.dynamic-import", + "py.monkeypatch": "python.monkeypatch", + "py.reflection": "python.reflection", + "py.star_import": "python.star-import", + "py.metaclass": "python.metaclass", + "py.syntax_error": "source.malformed-source", + "ts.module": "typescript.file", + "ts.directory_module": "typescript.module", + "ts.function": "typescript.function", + "ts.async_function": "typescript.function", + "ts.class": "typescript.class", + "ts.method": "typescript.method", + "ts.interface": "typescript.interface", + "ts.type": "typescript.type", + "ts.enum": "typescript.enum", + "ts.const": "typescript.const", + "ts.import": "typescript.import", + "ts.import_alias": "typescript.import", + "ts.export": "typescript.export", + "ts.reexport": "typescript.export", + "ts.route": "typescript.route", + "ts.dynamic_import": "typescript.dynamic-import", + "ts.decorator": "typescript.decorator", + "ts.namespace": "typescript.namespace", + "ts.commonjs_require": "typescript.commonjs-require", + "ts.ambient_module": "typescript.ambient-module", + "ts.syntax_error": "source.malformed-source", + "src.repository": "source.repository", + "src.file": "source.file", + "src.empty_file": "source.empty-file", + "src.trailing_newline": "source.trailing-newline", + "src.binary_file": "source.binary-file", + "src.malformed_source": "source.malformed-source", + "src.large_file": "source.large-file", + "src.path_escape": "source.path-escape" } } diff --git a/apps/backend/tests/benchmark/loader.py b/apps/backend/tests/benchmark/loader.py index c1d9aaf..1cc6e68 100644 --- a/apps/backend/tests/benchmark/loader.py +++ b/apps/backend/tests/benchmark/loader.py @@ -7,7 +7,7 @@ unsupported schema versions; duplicate fixture ids; duplicate expected identities; missing source files; absolute paths; ``..`` escapes; invalid line -ranges; undeclared support-matrix construct ids; malformed expected facts; +ranges; undeclared benchmark construct ids; malformed expected facts; unsupported languages; inconsistent producer versions; facts missing mandatory evidence; and accidental machine-blessed output committed as source truth. @@ -18,12 +18,17 @@ from __future__ import annotations import json -from dataclasses import dataclass, field +from dataclasses import dataclass from fractions import Fraction from pathlib import Path from typing import Any, Mapping -from app.extraction.support_matrix import SUPPORT_MATRIX as PRODUCTION_SUPPORT_MATRIX +from app.extraction.support_matrix import ( + CAPABILITIES_BY_ID, + CONSTRUCT_CAPABILITIES, + SupportStatus, + validate_registry, +) from app.intelligence import canonical from benchmark import schema @@ -37,7 +42,23 @@ class ManifestError(ValueError): - """A fixture, support matrix, or thresholds file failed strict validation.""" + """A fixture, capability mapping, or thresholds file failed strict validation.""" + + +_BENCHMARK_LANGUAGE_BY_PREFIX = { + "py": "python", + "src": "source", + "ts": "typescript", +} +_CROSS_LANGUAGE_SOURCE_CONSTRUCTS = {"file", "malformed-source", "repository"} + + +def _benchmark_language(construct_id: str, *, path: Path) -> str: + prefix, separator, _ = construct_id.partition(".") + language = _BENCHMARK_LANGUAGE_BY_PREFIX.get(prefix) + if not separator or language is None: + raise ManifestError(f"{path}: construct {construct_id!r} has an unsupported benchmark id prefix") + return language # --------------------------------------------------------------------------- @@ -54,6 +75,7 @@ class ConstructSpec: expected_diagnostic: str | None matrix_language: str matrix_construct: str + capability_id: str @dataclass(frozen=True) @@ -88,73 +110,64 @@ def _read_json(path: Path) -> Any: def load_support_matrix(path: Path) -> SupportMatrix: data = _read_json(path) if data.get("schemaVersion") != schema.SUPPORT_MATRIX_SCHEMA_VERSION: - raise ManifestError( - f"{path}: unsupported support-matrix schema version {data.get('schemaVersion')!r}" - ) + raise ManifestError(f"{path}: unsupported capability-mapping schema version {data.get('schemaVersion')!r}") constructs: dict[str, ConstructSpec] = {} raw = data.get("constructs") if not isinstance(raw, dict) or not raw: raise ManifestError(f"{path}: 'constructs' must be a non-empty object") mappings = data.get("productionMappings") if not isinstance(mappings, dict) or set(mappings) != set(raw): - raise ManifestError( - f"{path}: productionMappings must map every benchmark construct exactly once" - ) - covered_production: set[tuple[str, str]] = set() + raise ManifestError(f"{path}: productionMappings must map every benchmark construct exactly once") + try: + validate_registry() + except ValueError as exc: + raise ManifestError(f"{path}: invalid production capability registry: {exc}") from exc + covered_capabilities: set[str] = set() for construct_id, spec in sorted(raw.items()): - language = spec.get("language") - if language not in schema.LANGUAGES: - raise ManifestError(f"{path}: construct {construct_id!r} has unsupported language {language!r}") - supported = spec.get("supported") - if not isinstance(supported, bool): - raise ManifestError(f"{path}: construct {construct_id!r} 'supported' must be boolean") - expected_diagnostic = spec.get("expectedDiagnostic") - if not supported and expected_diagnostic not in schema.DIAGNOSTIC_CODES: - raise ManifestError( - f"{path}: unsupported construct {construct_id!r} must declare a valid 'expectedDiagnostic'" - ) - if expected_diagnostic is not None and expected_diagnostic not in schema.DIAGNOSTIC_CODES: - raise ManifestError(f"{path}: construct {construct_id!r} has unknown diagnostic {expected_diagnostic!r}") - mapping = mappings[construct_id] - matrix_language = str(mapping.get("language", "")) - matrix_construct = str(mapping.get("construct", "")) - if matrix_language not in PRODUCTION_SUPPORT_MATRIX: + if not isinstance(spec, dict) or not isinstance(spec.get("description"), str) or not spec["description"]: + raise ManifestError(f"{path}: construct {construct_id!r} must have a description") + capability_id = mappings[construct_id] + if not isinstance(capability_id, str): + raise ManifestError(f"{path}: construct {construct_id!r} mapping must be a capability id string") + capability = CAPABILITIES_BY_ID.get(capability_id) + if capability is None: raise ManifestError( - f"{path}: construct {construct_id!r} maps to unknown production matrix {matrix_language!r}" + f"{path}: construct {construct_id!r} maps to unknown production capability {capability_id!r}" ) - production = PRODUCTION_SUPPORT_MATRIX[matrix_language] - production_supported = matrix_construct in production.supported - production_unsupported = matrix_construct in production.unsupported - if not (production_supported or production_unsupported): + if construct_id not in capability.benchmark_ids: raise ManifestError( - f"{path}: construct {construct_id!r} maps to unknown production construct " - f"{matrix_language}.{matrix_construct}" + f"{path}: construct {construct_id!r} is not declared by production capability {capability_id!r}" ) - if supported != production_supported: - raise ManifestError( - f"{path}: construct {construct_id!r} support status disagrees with " - f"{matrix_language}.{matrix_construct}" - ) - covered_production.add((matrix_language, matrix_construct)) + supported = capability.status == SupportStatus.SUPPORTED + expected_diagnostic = capability.expected_diagnostic + if expected_diagnostic is not None and expected_diagnostic not in schema.DIAGNOSTIC_CODES: + raise ManifestError(f"{path}: capability {capability_id!r} has unknown diagnostic {expected_diagnostic!r}") + covered_capabilities.add(capability_id) constructs[construct_id] = ConstructSpec( construct_id=construct_id, - language=language, + language=_benchmark_language(construct_id, path=path), supported=supported, description=str(spec.get("description", "")), expected_diagnostic=expected_diagnostic, - matrix_language=matrix_language, - matrix_construct=matrix_construct, + matrix_language=capability.language, + matrix_construct=capability.construct, + capability_id=capability.id, ) - missing_production = sorted( - (language, construct) - for language, matrix in PRODUCTION_SUPPORT_MATRIX.items() - for construct in (*matrix.supported, *matrix.unsupported) - if (language, construct) not in covered_production + missing_benchmark_ids = sorted( + benchmark_id + for capability in CONSTRUCT_CAPABILITIES + for benchmark_id in capability.benchmark_ids + if benchmark_id not in raw ) - if missing_production: + if missing_benchmark_ids: raise ManifestError( - f"{path}: production support constructs have no benchmark mapping: {missing_production}" + f"{path}: production capability benchmark ids have no benchmark mapping: {missing_benchmark_ids}" ) + missing_capabilities = sorted( + capability.id for capability in CONSTRUCT_CAPABILITIES if capability.id not in covered_capabilities + ) + if missing_capabilities: + raise ManifestError(f"{path}: production capabilities have no benchmark mapping: {missing_capabilities}") return SupportMatrix(constructs=constructs, note=str(data.get("note", ""))) @@ -377,9 +390,7 @@ def _build_fact(group: str, raw: dict[str, Any], *, where: str, producers: set[s raise ManifestError(f"{where}: unknown fact group {group!r}") -def _validate_evidence_against_source( - sources: Mapping[str, bytes], span: EvidenceSpan, *, where: str -) -> None: +def _validate_evidence_against_source(sources: Mapping[str, bytes], span: EvidenceSpan, *, where: str) -> None: """Enforce RFC-0001 §6.2: the cited file exists, decodes, and the span is in range.""" data = sources.get(span.path) @@ -459,7 +470,10 @@ def load_fixture(directory: Path, support_matrix: SupportMatrix) -> LoadedFixtur _require(isinstance(producer_list, list) and bool(producer_list), f"{where0}: producerVersionSet must be non-empty") for producer in producer_list: _require( - isinstance(producer, str) and "@" in producer and not producer.startswith("@") and not producer.endswith("@"), + isinstance(producer, str) + and "@" in producer + and not producer.startswith("@") + and not producer.endswith("@"), f"{where0}: producerVersionSet entry {producer!r} must be 'name@version'", ) producers = set(producer_list) @@ -473,9 +487,12 @@ def load_fixture(directory: Path, support_matrix: SupportMatrix) -> LoadedFixtur f"{where0}: maxSourceBytes must be a positive integer", ) for construct_id in constructs_covered: - _require(construct_id in support_matrix, f"{where0}: undeclared support-matrix construct {construct_id!r}") + _require(construct_id in support_matrix, f"{where0}: undeclared benchmark construct {construct_id!r}") + spec = support_matrix.constructs[construct_id] _require( - support_matrix.constructs[construct_id].language in (language, "mixed") or language == "mixed", + spec.language == language + or language == "mixed" + or (spec.language == "source" and spec.matrix_construct in _CROSS_LANGUAGE_SOURCE_CONSTRUCTS), f"{where0}: construct {construct_id!r} language mismatch with fixture language {language!r}", ) diff --git a/apps/backend/tests/benchmark/report.py b/apps/backend/tests/benchmark/report.py index 63592ed..8b8c955 100644 --- a/apps/backend/tests/benchmark/report.py +++ b/apps/backend/tests/benchmark/report.py @@ -130,6 +130,11 @@ def to_json_dict(report: BenchmarkReport) -> dict[str, object]: "uncoveredUnsupported": report.parity.uncovered_unsupported, }, "requiredDiagnostics": {"passed": report.diagnostics.passed, "missing": report.diagnostics.missing}, + "goldenRegression": { + "passed": report.golden_regression_passed, + "unexpectedFactCount": len(report.scoring.report.false_positives) if report.scoring.report else None, + "missingFactCount": len(report.scoring.report.false_negatives) if report.scoring.report else None, + }, "scoring": scoring, "failures": report.failures(), } @@ -217,6 +222,16 @@ def to_markdown(report: BenchmarkReport) -> str: f"{len(report.parity.uncovered_unsupported)} unsupported unexercised." ) + lines += ["", "## Exact golden regression gate", ""] + if report.scoring.report is None: + lines.append("**Unavailable:** no real extraction comparison was produced.") + else: + lines.append( + f"Exact committed-golden comparison: **{'pass' if report.golden_regression_passed else 'fail'}** " + f"({len(report.scoring.report.false_positives)} unexpected, " + f"{len(report.scoring.report.false_negatives)} missing facts)." + ) + lines += ["", "## Extraction quality (precision / recall)", ""] if not report.scoring.available or report.scoring.report is None: lines.append("**Unavailable:** the real extraction adapter did not produce measurements.") diff --git a/apps/backend/tests/benchmark/runner.py b/apps/backend/tests/benchmark/runner.py index d7e5aa3..88c25c3 100644 --- a/apps/backend/tests/benchmark/runner.py +++ b/apps/backend/tests/benchmark/runner.py @@ -7,7 +7,7 @@ - a manifest is invalid; - provenance validity is below threshold; - determinism fails; -- the fixture/support-matrix parity check fails; +- the fixture/capability-mapping parity check fails; - an expected (required) diagnostic is missing for a declared blind spot; - the real extraction adapter is unavailable; - precision or recall is below threshold; or @@ -111,6 +111,20 @@ def scoring_gate_passed(self) -> bool: and provenance_ok ) + @property + def golden_regression_passed(self) -> bool: + """Require an exact committed-golden match independent of thresholds. + + Thresholds remain useful quality summaries, but a small semantic drift + must not pass merely because aggregate precision/recall stays high. + An intentional change is therefore visible as a reviewed expectation + diff and a corresponding exact-golden baseline change. + """ + + if not self.scoring.available or self.scoring.report is None: + return False + return not self.scoring.report.false_positives and not self.scoring.report.false_negatives + @property def passed(self) -> bool: if self.load_error is not None: @@ -120,6 +134,7 @@ def passed(self) -> bool: and self.determinism_passed and self.parity.passed and self.diagnostics.passed + and self.golden_regression_passed and self.scoring_gate_passed ) @@ -146,6 +161,12 @@ def failures(self) -> list[str]: for construct in self.parity.uncovered_unsupported: reasons.append(f"parity: unsupported construct {construct!r} is not exercised by any fixture") reasons.extend(f"diagnostics: {item}" for item in self.diagnostics.missing) + if not self.golden_regression_passed and self.scoring.report is not None: + reasons.append( + "golden regression: exact committed expectations changed " + f"({len(self.scoring.report.false_positives)} unexpected, " + f"{len(self.scoring.report.false_negatives)} missing facts)" + ) if not self.scoring.available or self.scoring.report is None: reasons.append("scoring: the real extraction adapter did not produce measurements") elif not self.scoring_gate_passed: diff --git a/apps/backend/tests/benchmark/schema.py b/apps/backend/tests/benchmark/schema.py index 442ac06..10f156b 100644 --- a/apps/backend/tests/benchmark/schema.py +++ b/apps/backend/tests/benchmark/schema.py @@ -8,7 +8,7 @@ # Fixture manifest schema. Bumping this is a deliberate, reviewed migration. FIXTURE_SCHEMA_VERSION = "ri-benchmark.v1" -SUPPORT_MATRIX_SCHEMA_VERSION = "ri-benchmark-support-matrix.v1" +SUPPORT_MATRIX_SCHEMA_VERSION = "ri-benchmark-support-matrix.v2" THRESHOLDS_SCHEMA_VERSION = "ri-benchmark-thresholds.v1" FIXTURE_CLASSES = ("minimal", "realistic", "adversarial") diff --git a/apps/backend/tests/benchmark/test_loader.py b/apps/backend/tests/benchmark/test_loader.py index 132a0f6..841eea4 100644 --- a/apps/backend/tests/benchmark/test_loader.py +++ b/apps/backend/tests/benchmark/test_loader.py @@ -13,6 +13,7 @@ import pytest +from app.extraction.support_matrix import CONSTRUCT_CAPABILITIES, MANIFEST_CAPABILITIES from benchmark import paths from benchmark.loader import ( ManifestError, @@ -33,6 +34,15 @@ def test_committed_corpus_loads_cleanly(): assert len(ids) == len(set(ids)), "fixture ids must be unique" +def test_benchmark_mapping_is_complete_and_registry_authoritative(): + for capability in CONSTRUCT_CAPABILITIES: + for benchmark_id in capability.benchmark_ids: + assert benchmark_id in SUPPORT_MATRIX.constructs + assert SUPPORT_MATRIX.constructs[benchmark_id].capability_id == capability.id + for capability in MANIFEST_CAPABILITIES: + assert capability.benchmark_disclosure + + def test_thresholds_load_with_exact_fractions(): thresholds = load_thresholds(paths.THRESHOLDS_PATH) assert thresholds.provenance_validity == 1 @@ -76,7 +86,13 @@ def _base_manifest() -> dict: "stableKey": "repo:root", "name": "repository", "evidence": [ - {"path": "README.md", "startLine": 1, "endLine": 1, "extractor": "repository-inventory", "extractorVersion": "1.1.0"} + { + "path": "README.md", + "startLine": 1, + "endLine": 1, + "extractor": "repository-inventory", + "extractorVersion": "1.1.0", + } ], }, { @@ -86,7 +102,13 @@ def _base_manifest() -> dict: "language": "python", "constructs": ["py.function.def"], "evidence": [ - {"path": "src/a.py", "startLine": 1, "endLine": 1, "extractor": "python-ast", "extractorVersion": "1.0.0"} + { + "path": "src/a.py", + "startLine": 1, + "endLine": 1, + "extractor": "python-ast", + "extractorVersion": "1.0.0", + } ], }, ] @@ -123,6 +145,15 @@ def test_sanity_base_manifest_loads(tmp_path: Path): assert symbol.language == "python" +def test_source_policy_construct_cannot_bypass_fixture_language(tmp_path: Path): + manifest = _base_manifest() + manifest["constructsCovered"] = ["src.path_escape"] + directory = _write_fixture(tmp_path, manifest) + + with pytest.raises(ManifestError, match="language mismatch"): + load_fixture(directory, SUPPORT_MATRIX) + + def test_manifest_synthetic_sources_are_deterministic_and_do_not_need_host_paths(tmp_path: Path): manifest = _base_manifest() manifest["syntheticFiles"] = { @@ -166,11 +197,14 @@ def test_loader_rejects_invalid_synthetic_sources(tmp_path: Path, synthetic_file (lambda m: m.update(revisionIdentity="git-sha"), "unsupported revisionIdentity"), (lambda m: m.update(blessed=True), "forbidden key"), (lambda m: m.update(producerVersionSet=["badproducer"]), "must be 'name@version'"), - (lambda m: m.update(constructsCovered=["py.not_a_construct"]), "undeclared support-matrix construct"), + (lambda m: m.update(constructsCovered=["py.not_a_construct"]), "undeclared benchmark construct"), (lambda m: m["expected"]["nodes"][1]["evidence"][0].update(path="/etc/passwd"), "absolute or escapes"), (lambda m: m["expected"]["nodes"][1]["evidence"][0].update(path="../escape.py"), "absolute or escapes"), (lambda m: m["expected"]["nodes"][1]["evidence"][0].update(startLine=0), "start_line must be >= 1"), - (lambda m: m["expected"]["nodes"][1]["evidence"][0].update(startLine=2, endLine=1), "end_line must be >= start_line"), + ( + lambda m: m["expected"]["nodes"][1]["evidence"][0].update(startLine=2, endLine=1), + "end_line must be >= start_line", + ), (lambda m: m["expected"]["nodes"][1]["evidence"][0].update(endLine=999), "exceeds logical_line_count"), (lambda m: m["expected"]["nodes"][1]["evidence"][0].update(extractor="ghost"), "not in producerVersionSet"), (lambda m: m["expected"]["nodes"][1].__setitem__("evidence", []), "must carry >=1 evidence"), diff --git a/apps/backend/tests/benchmark/test_regression_failpath.py b/apps/backend/tests/benchmark/test_regression_failpath.py index af3066d..6ffae36 100644 --- a/apps/backend/tests/benchmark/test_regression_failpath.py +++ b/apps/backend/tests/benchmark/test_regression_failpath.py @@ -103,6 +103,7 @@ def test_a_silent_extractor_fails_the_recall_gate(): assert result.scoring.report.overall.recall < result.thresholds.recall assert not result.passed assert result.exit_code == 1 + assert not result.golden_regression_passed def test_an_inventing_extractor_fails_the_precision_gate(): @@ -111,6 +112,7 @@ def test_an_inventing_extractor_fails_the_precision_gate(): assert result.scoring.report.overall.precision < result.thresholds.precision assert not result.passed assert result.exit_code == 1 + assert not result.golden_regression_passed def test_an_invalid_emitted_citation_fails_even_if_scoring_is_high(): @@ -134,14 +136,14 @@ def test_a_broken_manifest_fails_the_build(tmp_path: Path): def test_a_support_matrix_mismatch_fails_the_build(tmp_path: Path): source = runner.paths.SUPPORT_MATRIX_PATH matrix = json.loads(source.read_text(encoding="utf-8")) - matrix["productionMappings"]["py.function.def"]["construct"] = "not-a-real-construct" + matrix["productionMappings"]["py.function.def"] = "not-a-real-capability" matrix_path = tmp_path / "support-matrix.json" matrix_path.write_text(json.dumps(matrix), encoding="utf-8") result = runner.run(support_matrix_path=matrix_path) assert result.load_error is not None - assert "unknown production construct" in result.load_error + assert "unknown production capability" in result.load_error assert not result.passed assert result.exit_code == 1 diff --git a/apps/backend/tests/benchmark/test_runner.py b/apps/backend/tests/benchmark/test_runner.py index 0d0e5ba..2c62170 100644 --- a/apps/backend/tests/benchmark/test_runner.py +++ b/apps/backend/tests/benchmark/test_runner.py @@ -28,6 +28,7 @@ def test_full_benchmark_passes_and_reports_all_real_signals(): assert result.scoring.report.overall.recall == 1 assert result.scoring.actual_provenance is not None assert result.scoring.actual_provenance.validity == 1 + assert result.golden_regression_passed # All three fixture classes and both languages are represented. assert set(result.corpus.by_class) == {"minimal", "realistic", "adversarial"} assert {"python", "typescript"} <= set(result.corpus.by_language) @@ -41,6 +42,7 @@ def test_precision_recall_is_measured_by_the_real_extractors(): assert result.scoring.report.false_positives == [] assert result.scoring.report.false_negatives == [] assert result.scoring_gate_passed + assert result.golden_regression_passed def test_reports_are_deterministic_and_serialisable(): @@ -60,6 +62,7 @@ def test_reports_are_deterministic_and_serialisable(): assert "class:adversarial" in markdown assert "False positives" in markdown assert "False negatives" in markdown + assert "Exact golden regression gate" in markdown assert "deferred" not in markdown.lower() assert "unavailable" not in markdown.lower() diff --git a/apps/backend/tests/extraction/test_capability_registry.py b/apps/backend/tests/extraction/test_capability_registry.py new file mode 100644 index 0000000..a08c9ac --- /dev/null +++ b/apps/backend/tests/extraction/test_capability_registry.py @@ -0,0 +1,74 @@ +"""Focused tests for the authoritative capability registry and README view.""" + +from __future__ import annotations + +from dataclasses import replace +from pathlib import Path + +import pytest + +from app.extraction.support_matrix import ( + CAPABILITY_REGISTRY, + PUBLIC_CAPABILITIES, + README_CAPABILITIES_END, + README_CAPABILITIES_START, + PublicStatus, + SupportStatus, + check_readme_capabilities, + render_readme_capabilities, + validate_registry, +) + + +def test_registry_is_typed_unique_and_deterministically_ordered(): + validate_registry() + ids = [item.id for item in CAPABILITY_REGISTRY] + assert ids == sorted(ids) + assert len(ids) == len(set(ids)) + assert all(isinstance(item.status, SupportStatus) for item in CAPABILITY_REGISTRY) + + +def test_registry_rejects_duplicate_ids_and_unknown_status(): + with pytest.raises(ValueError, match="duplicate capability ids"): + validate_registry((CAPABILITY_REGISTRY[0], CAPABILITY_REGISTRY[0])) + + broken = CAPABILITY_REGISTRY[0] + broken = broken.__class__( + id=broken.id, + language=broken.language, + construct=broken.construct, + status="unknown", # type: ignore[arg-type] + description=broken.description, + limitation=broken.limitation, + benchmark_ids=broken.benchmark_ids, + ) + with pytest.raises(ValueError, match="unsupported status"): + validate_registry((broken,)) + + +def test_public_claims_resolve_to_consistent_registry_capabilities(): + validate_registry() + assert all(isinstance(item.status, PublicStatus) for item in PUBLIC_CAPABILITIES) + + broken = replace(PUBLIC_CAPABILITIES[0], capability_ids=("product.missing",)) + with pytest.raises(ValueError, match="references unknown capability"): + validate_registry(public_capabilities=(broken,)) + + +def test_readme_capability_rendering_is_byte_stable_and_checked_in(): + assert render_readme_capabilities() == render_readme_capabilities() + readme = Path(__file__).parents[4] / "README.md" + check_readme_capabilities(readme) + block = readme.read_text(encoding="utf-8") + assert block.count(README_CAPABILITIES_START) == 1 + assert block.count(README_CAPABILITIES_END) == 1 + + +def test_stale_readme_capability_claims_fail_check(tmp_path: Path): + readme = tmp_path / "README.md" + content = (Path(__file__).parents[4] / "README.md").read_text(encoding="utf-8") + readme.write_text( + content.replace("Archive upload and public GitHub import", "Changed capability", 1), encoding="utf-8" + ) + with pytest.raises(ValueError, match="README capability registry block is stale"): + check_readme_capabilities(readme) diff --git a/docs/architecture/REPOSITORY_INTELLIGENCE.md b/docs/architecture/REPOSITORY_INTELLIGENCE.md index 5046506..6ccb286 100644 --- a/docs/architecture/REPOSITORY_INTELLIGENCE.md +++ b/docs/architecture/REPOSITORY_INTELLIGENCE.md @@ -81,8 +81,17 @@ dispatching those bytes through each extractor's real `supports()` method. `PythonExtractor` uses Python's AST and `TypeScriptExtractor` uses tree-sitter. Both emit normalized nodes, observations, diagnostics, and line evidence through the `ExtractionResult` contract. Their declared support and blind spots live in -`app/extraction/support_matrix.py`. Durable analysis stores their output in a -sealed normalized snapshot. Historical legacy JSON is not produced or consumed. +the typed capability registry in `app/extraction/support_matrix.py`; the legacy +`SUPPORT_MATRIX` import is only a derived compatibility view. Durable analysis +stores their output in a sealed normalized snapshot. Historical legacy JSON is +not produced or consumed. + +The registry is the authoritative source for construct status, limitations, +stable ids, benchmark mappings, supported dependency-manifest names, and the +public capability assessments rendered in the README. Run +`python scripts/check-capabilities.py` to validate registry structure, +registry-to-benchmark parity, deterministic rendering, and the committed +README block. The command checks only; it never rewrites documentation. --- @@ -280,7 +289,7 @@ because provider prose cannot be deterministically mapped to stored facts. - **Revision identity:** first-class and immutable per imported repository revision. Snapshot history can be retained, but diff/query APIs and product re-analysis orchestration are not implemented. - **Dependencies:** three manifest formats, no lockfiles, no transitive resolution, and no vulnerability or outdated-version scanning. The dependency API reports both assessments as explicit `not_computed` statuses; it emits no clean result or count without a scanner. - **Dependency inventory:** only direct declarations from accepted `package.json`, `pyproject.toml`, and `requirements.txt` paths are reported. The parser inventory excludes `.git`, dependency/install directories, build output, virtual environments, caches, vendor paths, and generated paths; lockfiles are not read. Each candidate is size-checked and read with the existing 512 KiB source budget before being processed individually; oversized manifests produce `RI-LIMIT-SKIP` rather than being retained in memory. Multiple workspace declarations remain attached to one logical dependency, including conflicts rather than an arbitrarily selected version. A malformed supported manifest produces a safe `RI-SRC-MALFORMED` diagnostic while valid manifests continue to contribute declarations. No transitive resolution, vulnerability scanning, or outdated-version scanning is implemented. `AnalysisWorker` merges same-producer dependency nodes that share a stable key into one node with a `declarations` list before persistence, specifically so a package declared in more than one manifest — an ordinary monorepo shape — does not fail sealing for the whole repository. -- **Languages:** meaningful extraction covers Python and TypeScript/JavaScript. Other languages get file-tree and metadata treatment only. +- **Languages:** the capability registry declares meaningful extraction for Python and TypeScript/JavaScript constructs. Other languages get file-tree and metadata treatment only. - **File size cap:** files over 512 KB are read as empty during extraction, so their contents contribute nothing. - **Build cost:** the whole repository is re-analysed from scratch in a background job; incremental analysis is not implemented. @@ -290,7 +299,7 @@ because provider prose cannot be deterministically mapped to stored facts. 1. Add observed syntax facts to the shared extraction contract and the relevant extractor; represent heuristic conclusions as explicit inferred assertions. -2. Update the published support matrix and focused extractor tests. +2. Update the authoritative capability registry and focused extractor tests. 3. Add or update an independently authored benchmark fixture and golden manifest. 4. Consume the normalized fact only through the Repository Intelligence query boundary once that boundary supports it. diff --git a/scripts/check-capabilities.py b/scripts/check-capabilities.py new file mode 100644 index 0000000..f580dab --- /dev/null +++ b/scripts/check-capabilities.py @@ -0,0 +1,38 @@ +"""Validate the production capability registry and its checked-in README view.""" + +from __future__ import annotations + +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "apps/backend")) +sys.path.insert(0, str(ROOT / "apps/backend/tests")) + +from app.extraction.support_matrix import ( # noqa: E402 + check_readme_capabilities, + render_readme_capabilities, + validate_registry, +) +from benchmark.loader import load_support_matrix # noqa: E402 +from benchmark.paths import SUPPORT_MATRIX_PATH # noqa: E402 + + +def main() -> int: + validate_registry() + first = render_readme_capabilities() + second = render_readme_capabilities() + if first != second: + raise SystemExit("capability README rendering is not deterministic") + check_readme_capabilities(ROOT / "README.md") + matrix = load_support_matrix(SUPPORT_MATRIX_PATH) + if not matrix.constructs: + raise SystemExit("benchmark capability mapping is empty") + print(f"Capability registry valid: {len(matrix.constructs)} benchmark mappings") + print("README capability registry is current and deterministic") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())