chore(intelligence): add capability registry golden gate - #227
Conversation
There was a problem hiding this comment.
Review — changes requested
Right lane and the design is genuinely good: a typed registry, benchmark IDs
demoted to fixture taxonomy, and an exact-golden gate layered over the aggregate
thresholds. §23 W0 names "capability registry" as an increment and §24 GATE A
lists it in FOUNDATION, so this is a precondition for all W1 breadth work — worth
getting exactly right rather than merged quickly.
Four items, roughly in the order I'd tackle them.
1. The README table isn't derived from the registry
This is the core of the PR and the W0 exit criterion — "every shipped claim maps
to tested fact/evidence contracts" — and issue #208 AC 3 restates it: README
claims must derive from the registry "so they cannot drift from code."
PUBLIC_CAPABILITIES is a hand-written prose tuple that shares a file with
CAPABILITY_REGISTRY but has no linkage to it:
- no
PublicCapability.id(dependency-graph,archive-import, …) resolves to
anyCapability.id validate_registry()contains zero references toPUBLIC_CAPABILITIESPublicCapability.statusis a barestr, so the vocabulary is unvalidated
Concretely: the dependency-graph row hardcodes "Direct declarations from
package.json, requirements.txt, and pyproject.toml" in prose, while
MANIFEST_CAPABILITIES is the real source that supported_manifest_filenames()
reads. Add a fourth manifest capability and the extractor picks it up, the prose
does not, and check_readme_capabilities() still passes — because it only
compares README against render_readme_capabilities(), which reads that same
stale constant.
So the gate proves README matches a constant, not that it matches code. Please
extend validate_registry() to assert every PublicCapability resolves to
registry capabilities whose statuses are consistent with its claim.
2. The status vocabulary doesn't match §01
§01's Truth boundary is explicit: the four terms are implemented,
implemented with disclosed limits, planned, and rejected, and
"marketing copy, API responses, and UI states must use the same vocabulary." The
README is marketing copy.
Partially implemented isn't a roadmap term, and Not implemented / not assessed
collapses two distinct states. That matters here: revision-comparison carries it
even though §01 calls evolution intelligence "the primary strategic build," and
dependency-scanning carries it although §27 doesn't reject scanning. Both are
planned, and the current wording reads as though they may never ship.
Making PublicCapability.status an enum over §01's four terms fixes this and
item 1's validation gap in one move.
3. Rebase on dev and run ruff format
#230 merged, so the backend CI now runs ruff check / ruff format --check /
mypy. Current state of this branch against the new dev:
| Check | Result |
|---|---|
| Merge | conflicts in .github/workflows/ci.yml, app/extraction/support_matrix.py |
ruff check |
pass |
ruff format --check |
fail — support_matrix.py |
mypy |
pass (support_matrix.py isn't in the ignore list and is clean) |
The single-line _capability(...) entries are what the formatter rejects. After
ruff format they wrap, which reads better anyway.
4. Narrow the loader escape hatch
loader.py now accepts any construct where matrix_language == "source". That's
needed because py.syntax_error / ts.syntax_error map to
source.malformed-source, so the derived ConstructSpec.language is no longer
python/typescript — but it's unscoped, and disables the fixture-language check
for all eight source.* constructs on any fixture. A Python fixture can now
declare src.path_escape silently. Keep the benchmark's own language on
ConstructSpec (derived from the ID prefix) and validate against that.
Smaller
supported_manifest_filenames()is called inside
DependencyManifestExtractor.supports(), which runs once per repository file,
and does asorted()+ genexpr +removeprefixeach call.dependency_graph.py
already binds it at module level — do the same here, or@lru_cache.- Keep the precision/recall thresholds. §15 requires them published per language
with disclosure, and §16's Accuracy gate ("regressions are triaged, not averaged
away") is exactly the argument for layering the exact-golden gate on top. The
layering is correct as built. validate_registry()at import time turns a malformed registry into an
app-startupImportError, sincedependency_graphandmanifestsnow import
this module. Deliberate fail-fast, but worth a comment saying so.
Governance note
This touches two §25 Controlled-lane artifacts — public claims (all 16 statements)
and a schema version bump (ri-benchmark-support-matrix v1 → v2). Both need
explicit sign-off on wording and on the bump, separate from code review.
Please also fill in Roadmap alignment
CONTRIBUTING.md §0 makes it a merge gate and the section is missing. It maps
cleanly:
- §23 workstream: W0 / Truth contract — "capability registry" is a named increment
- §24 gate: GATE A, FOUNDATION (golden corpus + capability registry)
- §16 gate: Accuracy gate
- §28 criterion: Trusted output
5df45d9 to
a440136
Compare
|
Approved. Solid additive governance: the capability registry + README validation gate is exactly the transparent-evidence work we want (§08), and it stays out of the runtime extraction path so it can't break pipelines. Tests cover the registry and benchmark. Nice. |
parthrohit22
left a comment
There was a problem hiding this comment.
Approved. Capability registry + CI validation gate, additive and out of runtime path. Tests present.
Summary
Why
Issue #208 requires one auditable capability source of truth, explicit limitations and diagnostics, deterministic benchmark mappings, and a drift gate that prevents silent changes to the committed golden corpus.
Validation
python scripts/check-capabilities.pyruff check appruff format --check appmypy app: no issues in 117 source filesRoadmap alignment
Governance sign-off required
ri-benchmark-support-matrixv1 → v2 schema bumpRelated to #208