docs(recon): state-of-codebase report for v022-polish show

_recon/state.md

@@ -0,0 +1,69 @@
+# State of the Codebase
+
+This synthesis is based only on the upstream explorer artifacts requested for this round. The dependency graph scanner found a 25-member Rust workspace rooted at `crates/Cargo.toml`, plus a `khive-merge` crate directory that exists but is explicitly excluded from workspace membership (`../e1/dep_graph.md:7`-`../e1/dep_graph.md:10`). The reported crate table is the layering spine for this state report: foundation crates include `khive-types` and `khive-score`; platform includes `khive-storage`, `khive-db`, and `khive-query`; services include `khive-runtime`, `khive-request`, `khive-vcs`, and several packs; apps include `khive-mcp`, while several crates remain unassigned in the explorer graph (`../e1/dep_graph.md:207`-`../e1/dep_graph.md:234`).
+
+## 1. Orphan Crates
+
+The six requested retrieval/VCS/merge crates split into two categories. `khive-bm25`, `khive-hnsw`, and `khive-fusion` are consumed, but only through `khive-retrieval`: the reverse-dependency evidence points to `crates/khive-retrieval/Cargo.toml:15`, `:14`, and `:16` respectively (`../e1/orphan_crates.md:16`-`../e1/orphan_crates.md:18`). The scanner also found non-manifest source use inside `khive-retrieval`, including BM25 persistence imports/re-exports, HNSW persistence imports/re-exports, and fusion imports in the hybrid search surface (`../e1/orphan_crates.md:50`-`../e1/orphan_crates.md:58`).
+
+`khive-retrieval`, `khive-vcs-adapters`, and `khive-merge` have zero reverse manifest dependencies under `crates/*/Cargo.toml` (`../e1/orphan_crates.md:19`-`../e1/orphan_crates.md:21`). In the full workspace-wide zero-reverse-dep list, the same condition also applies to `khive-gate-rego`, `khive-pack-template`, `khive-mcp`, `kkernel`, `khive-retrieval`, and the non-workspace `khive-merge` (`../e1/dep_graph.md:236`-`../e1/dep_graph.md:246`). Not all zero-reverse crates are dead code: `khive-mcp` and `kkernel` are binary targets, with explicit bin target evidence in their manifests (`../e1/orphan_crates.md:10`, `../e1/dep_graph.md:229`, `../e1/dep_graph.md:232`). The architectural risk is concentrated in non-binary zero-reverse crates: `khive-vcs-adapters` has no test files and no reverse deps (`../e3/test_inventory.md:211`, `../e1/orphan_crates.md:31`), while `khive-retrieval` has a large public API surface but no downstream manifest consumer (`../e3/api_surface.md:22`, `../e1/dep_graph.md:233`).
+
+## 2. ADR Alignment Audit
+
+ADR-011 is mostly aligned on the runtime embedding boundary: `khive-runtime` depends on `lattice-embed`, stores one `OnceCell<Arc<dyn EmbeddingService>>`, and lazy-loads `NativeEmbeddingService` through `CachedEmbeddingService` (`../e2/adr_alignment.md:17`). The explorer also verified that runtime `embed()` and `embed_batch()` call the lattice service directly, with no separate khive-side embedding trait in the runtime path (`../e2/adr_alignment.md:18`). The notable ADR-011 architecture gap is not a contradiction inside the boundary itself; it is that `khive-retrieval` is acting as a facade/re-export layer but no workspace crate depends on it today (`../e2/adr_alignment.md:21`, `../e1/dep_graph.md:233`).
+
+ADR-021 and ADR-033 are partly implemented in the memory pack. `MemoryPack` registers `memory`, `remember`, and `recall`; `remember` validates `source_id` as a UUID and links through `annotates`; recall filters alive memory notes, scores, sorts, truncates, and returns results (`../e2/adr_alignment.md:28`-`../e2/adr_alignment.md:31`). Configurable recall fields and per-call overrides are present, including subhandlers such as `recall.embed`, `recall.candidates`, `recall.fuse`, `recall.rerank`, and `recall.score` (`../e2/adr_alignment.md:40`-`../e2/adr_alignment.md:42`). The drift is in registry semantics: accepted ADR text permits multiple memory pack instances and pack kind routing, but runtime rejects duplicate note kinds during registry build (`../e2/adr_alignment.md:33`, `../e2/adr_alignment.md:68`). Subhandler visibility is also drifted: subhandlers are excluded from catalog, but registry and MCP request dispatch still route arbitrary parsed tool names, including dotted subhandlers (`../e2/adr_alignment.md:43`, `../e2/adr_alignment.md:69`). Some memory gaps are explicitly phased out rather than regressions: rerank is present as a pass-through subhandler until a reranker model is wired, and engine/model multi-embedding behavior remains tied to deferred registry work (`../e2/adr_alignment.md:34`, `../e2/adr_alignment.md:45`-`../e2/adr_alignment.md:46`).
+
+ADR-043 is only partially landed. The `_embedding_models` DDL and event enum additions exist (`../e2/adr_alignment.md:52`-`../e2/adr_alignment.md:54`), but startup backfill, sqlite-vec rebuild/new-table creation, migration worker, drift integration, and useful `engine list/status/migrate/drift-check` behavior are follow-up work rather than complete runtime behavior (`../e2/adr_alignment.md:53`, `../e2/adr_alignment.md:55`-`../e2/adr_alignment.md:57`). One concrete regression remains: the ADR says the schema owner is `khive-runtime`, while the implementation currently lives in `khive-db` migrations (`../e2/adr_alignment.md:58`).
+
+## 3. Marketplace Plugins
+
+The KG plugin exposes 14 verbs: `create`, `get`, `list`, `update`, `delete`, `merge`, `search`, `link`, `neighbors`, `traverse`, `query`, plus ADR-046 proposal verbs `propose`, `review`, and `withdraw` (`../e2/marketplace_audit.md:14`-`../e2/marketplace_audit.md:27`). GTD exposes five verbs: `assign`, `next`, `complete`, `tasks`, and `transition` (`../e2/marketplace_audit.md:28`-`../e2/marketplace_audit.md:32`). The explorer found no removed KG or GTD verbs still referenced by the scanned marketplace skill files (`../e2/marketplace_audit.md:63`-`../e2/marketplace_audit.md:68`).
+
+Marketplace drift is documentation and coverage drift rather than broken verb references. Several KG skills still describe six entity kinds or thirteen relations, while current KG/type code exposes eight entity kinds and fifteen edge relations (`../e2/marketplace_audit.md:54`-`../e2/marketplace_audit.md:59`). The new KG proposal verbs have no corresponding KG marketplace skills: `propose`, `review`, and `withdraw` are exposed in code, but the scanned KG skill set is limited to `connect`, `digest`, `expand`, `explore`, `gap`, and `polish` (`../e2/marketplace_audit.md:70`-`../e2/marketplace_audit.md:76`). GTD skill examples were not flagged for stale statuses or priorities (`../e2/marketplace_audit.md:60`-`../e2/marketplace_audit.md:61`, `../e2/marketplace_audit.md:77`).
+
+## 4. Open Issues
+
+The open-issue inventory reports 64 open issues from `gh issue list --state open --limit 200 --json number,title,labels`, with category counts of 16 bug, 4 docs, 7 feature, 22 other, and 15 wiring (`../e3/open_issues.md:1`-`../e3/open_issues.md:8`). The same scan ranks 5 issues P0, 15 P1, 19 P2, and 25 P3 (`../e3/open_issues.md:9`-`../e3/open_issues.md:12`). The P0 set is not primarily the orphan/retrieval work: it includes three security/authorization/proposal isolation bugs plus `khive-retrieval` feature-gated test failures and `khive-hnsw` arena pointer safety (`../e3/open_issues.md:19`-`../e3/open_issues.md:21`, `../e3/open_issues.md:55`-`../e3/open_issues.md:56`).
+
+Many architecture gaps already have issue coverage. ADR-043 migration and drift work appears as #380 and #385 (`../e3/open_issues.md:29`, `../e3/open_issues.md:32`). Embedder registry work appears as #397 (`../e3/open_issues.md:18`). Memory recall gaps appear as #375, #376, #377, and #378 (`../e3/open_issues.md:34`-`../e3/open_issues.md:37`). VCS adapter scaffolding is tracked by #366 (`../e3/open_issues.md:42`). Marketplace/skill gaps are less directly covered by the listed issues; the closest open docs issue is brain pack discoverability rather than KG proposal skill coverage (`../e3/open_issues.md:61`, `../e2/marketplace_audit.md:74`-`../e2/marketplace_audit.md:76`).
+
+## 5. Embedding Surface
+
+The current embedding surface supports selecting one configured model, not multi-loaded runtime embedding. The workspace pins `lattice-embed = "0.2.4"`, and `khive-runtime`, `khive-retrieval`, and `khive-hnsw` all depend on it (`../e1/embedding_surface.md:7`-`../e1/embedding_surface.md:11`). `lattice-embed` itself includes both MiniLM and paraphrase MiniLM as 384-dimensional local models, with parsing aliases for both (`../e1/embedding_surface.md:20`-`../e1/embedding_surface.md:27`). It also has dual-write and migration primitives such as `EmbeddingRoutingConfig.write_models`, `BackfillConfig.dual_write`, and source/target routing (`../e1/embedding_surface.md:46`-`../e1/embedding_surface.md:55`).
+
+Khive runtime does not yet expose that as end-to-end dual embedding. `RuntimeConfig` has one `Option<EmbeddingModel>`, `KhiveRuntime` stores one embedder `OnceCell`, and `vectors()` opens one model-keyed vector namespace (`../e1/embedding_surface.md:62`-`../e1/embedding_surface.md:68`, `../e1/embedding_surface.md:123`-`../e1/embedding_surface.md:125`). `embed()` and `embed_batch()` call one service for one model, entity/note writes embed once, and memory recall embeds the query once against one vector store (`../e1/embedding_surface.md:69`-`../e1/embedding_surface.md:76`, `../e1/embedding_surface.md:126`-`../e1/embedding_surface.md:129`). The capability matrix therefore marks "dual embedding MiniLM + paraphrase end-to-end" as a gap (`../e1/embedding_surface.md:117`). Registry listing is scaffolded but empty: `kkernel engine list` calls a query function that currently returns `Ok(Vec::new())`, while migrate and drift-check return not implemented (`../e1/embedding_surface.md:99`-`../e1/embedding_surface.md:103`, `../e1/embedding_surface.md:130`-`../e1/embedding_surface.md:132`).
+
+## 6. Test Inventory
+
+The test inventory is broad but uneven. It found 33 explicit unit test files, 18 integration files, 1 e2e file, 1 smoke file, 122 inline `cfg(test)` modules, and no `benches` directory (`../e3/test_inventory.md:214`-`../e3/test_inventory.md:222`). CLI coverage is substantial for `kg` subcommands and CLI helper libraries (`../e3/test_inventory.md:4`-`../e3/test_inventory.md:29`). Rust integration tests exist for `khive-db`, `khive-mcp`, pack crates such as KG/GTD/Memory, `khive-runtime`, `khive-types`, and `khive-vcs` (`../e3/test_inventory.md:31`-`../e3/test_inventory.md:54`).
+
+The missing coverage aligns with the highest-risk scaffolding. `khive-vcs-adapters` has no tests found (`../e3/test_inventory.md:211`) and no reverse deps (`../e1/orphan_crates.md:31`). `khive-retrieval` has many unit tests but no crate-level integration test file (`../e3/test_inventory.md:205`), while open issue #309 reports feature-gated test failures in `khive-retrieval` (`../e3/open_issues.md:55`). `khive-merge` is non-workspace and unit-only, with no integration test file under crate tests or integration_tests (`../e3/test_inventory.md:195`, `../e1/orphan_crates.md:32`). The registry/dispatch ADR drift sits in crates that do have tests, especially `khive-runtime` and `khive-mcp`, so the gap is not absence of any tests but absence of targeted regression checks for duplicate note-kind routing and subhandler dispatch enforcement (`../e2/adr_alignment.md:68`-`../e2/adr_alignment.md:70`, `../e3/test_inventory.md:194`, `../e3/test_inventory.md:206`).
+
+## 7. CLI Behavior
+
+The TypeScript CLI entry point is `cli/main.ts`, with the Deno test task declared in `cli/deno.json`; npm shims exist for `khive` and `khive-mcp` (`../e3/cli_behavior.md:1`-`../e3/cli_behavior.md:6`). The main CLI has groups for `kg`, `auth`, and `pack` (`../e3/cli_behavior.md:11`-`../e3/cli_behavior.md:43`). Most implemented `kg` commands have direct tests: `init`, `validate`, `hook`, `config`, `embed`, `resolve`, `migrate`, `export`, `import`, `diff`, `log`, `stats`, and `doctor` have cited tests (`../e3/cli_behavior.md:13`-`../e3/cli_behavior.md:28`). `kg commit`, `kg sync`, and `kg status` are implemented but have no direct command tests (`../e3/cli_behavior.md:14`-`../e3/cli_behavior.md:16`, `../e3/cli_behavior.md:84`-`../e3/cli_behavior.md:86`).
+
+The CLI also exposes deferred or not-implemented surfaces. `kg update` is parsed but not implemented, `auth login/status/logout` are phase C2 and not implemented, and most `pack` commands beyond `pack init` and `pack check` are deferred (`../e3/cli_behavior.md:29`-`../e3/cli_behavior.md:43`). `pack check` has library validation tests but no direct `runPackCheck` command test (`../e3/cli_behavior.md:36`, `../e3/cli_behavior.md:92`). Rust binary targets are `khive-mcp` and `kkernel` (`../e3/cli_behavior.md:44`-`../e3/cli_behavior.md:48`), and the `kkernel engine` surface is present but scaffolded for ADR-043 (`../e1/embedding_surface.md:99`-`../e1/embedding_surface.md:103`).
+
+## Cross-Axis Findings
+
+The orphan-crate picture explains only part of the ADR drift. `khive-retrieval` being unconsumed explains why dual-index and retrieval facade primitives can exist without influencing runtime behavior (`../e1/dep_graph.md:233`, `../e1/embedding_surface.md:83`-`../e1/embedding_surface.md:89`, `../e1/embedding_surface.md:132`). It does not explain duplicate note-kind rejection, subhandler dispatch visibility, or ADR-043 schema ownership drift, which are in `khive-runtime`, `khive-mcp`, `khive-request`, and `khive-db` paths (`../e2/adr_alignment.md:58`, `../e2/adr_alignment.md:68`-`../e2/adr_alignment.md:70`).
+
+Missing tests correlate with dormant or scaffolded surfaces more than with all drift. `khive-vcs-adapters` has both zero reverse deps and zero tests (`../e1/orphan_crates.md:31`, `../e3/test_inventory.md:211`). `khive-retrieval` has unit coverage but no integration coverage and is tied to open feature-gated failures (`../e3/test_inventory.md:205`, `../e3/open_issues.md:55`). By contrast, runtime and MCP have tests but still exhibit ADR drift, so those areas need targeted regression additions rather than generic test creation (`../e3/test_inventory.md:194`, `../e3/test_inventory.md:206`, `../e2/adr_alignment.md:68`-`../e2/adr_alignment.md:70`).
+
+Open issues already track much of the high-value migration work. #397 tracks EmbedderRegistry implementation, #380 tracks `engine migrate` and queueing, #385 tracks ADR-043 startup backfill, and #375-#378 track recall pipeline gaps (`../e3/open_issues.md:18`, `../e3/open_issues.md:29`, `../e3/open_issues.md:32`, `../e3/open_issues.md:34`-`../e3/open_issues.md:37`). New tracking is most useful for KG marketplace proposal skills and for explicit runtime/MCP visibility regression tests, because those gaps are visible in the explorer artifacts but not directly represented in the issue table (`../e2/marketplace_audit.md:74`-`../e2/marketplace_audit.md:76`, `../e2/adr_alignment.md:69`).
+
+## Prioritized Backlog
+
+- P0: Address security issues #394, #395, and #396 before broad architecture migration, because the issue inventory ranks them P0 and they affect authorization/proposal isolation/state pollution (`../e3/open_issues.md:19`-`../e3/open_issues.md:21`).
+- P0: Stabilize retrieval/HNSW foundations by resolving #309 and #308, then decide whether `khive-retrieval` is a real consumed facade or a staged crate. This touches unconsumed retrieval architecture plus P0 test/safety issues (`../e3/open_issues.md:55`-`../e3/open_issues.md:56`, `../e1/dep_graph.md:233`).
+- P1: Fix ADR-017/021 duplicate note-kind semantics or amend the ADRs. The current runtime rejects duplicate note kinds while accepted ADR text describes multi-instance kind routing (`../e2/adr_alignment.md:33`, `../e2/adr_alignment.md:68`).
+- P1: Enforce subhandler visibility at registry/MCP dispatch boundaries and add targeted runtime/MCP regression tests (`../e2/adr_alignment.md:43`, `../e2/adr_alignment.md:69`, `../e3/test_inventory.md:194`, `../e3/test_inventory.md:206`).
+- P1: Implement ADR-043 registry plumbing in dependency order: query `_embedding_models`, make `kkernel engine list/status` real, then implement migrate/drift-check follow-ups (#397, #380, #385) (`../e1/embedding_surface.md:99`-`../e1/embedding_surface.md:103`, `../e3/open_issues.md:18`, `../e3/open_issues.md:29`, `../e3/open_issues.md:32`).
+- P1: Convert one-model runtime embedding into a multi-engine/model registry path before claiming dual MiniLM + paraphrase support (`../e1/embedding_surface.md:117`, `../e1/embedding_surface.md:123`-`../e1/embedding_surface.md:132`).
+- P2: Refresh KG marketplace taxonomy text and add KG proposal skills for `propose`, `review`, and `withdraw` (`../e2/marketplace_audit.md:54`-`../e2/marketplace_audit.md:59`, `../e2/marketplace_audit.md:74`-`../e2/marketplace_audit.md:76`).
+- P2: Add direct CLI tests for implemented but uncovered commands: `khive`, `kg commit`, `kg sync`, `kg status`, and `pack check` (`../e3/cli_behavior.md:83`-`../e3/cli_behavior.md:92`).
+- P3: Remove, integrate, or explicitly park zero-reverse scaffolds such as `khive-vcs-adapters`, `khive-pack-template`, and non-workspace `khive-merge` after higher-risk runtime and embedding work is complete (`../e1/dep_graph.md:240`-`../e1/dep_graph.md:246`).
+
+Domain utility: SKIPPED - the requested lore suggest/compose tooling is not available in this session, so synthesis relied on upstream explorer artifacts and their citations.