diff --git a/.erpaval/INDEX.md b/.erpaval/INDEX.md index 854ed17b..4d13f79f 100644 --- a/.erpaval/INDEX.md +++ b/.erpaval/INDEX.md @@ -3,6 +3,10 @@ Compound-extracted lessons and EARS specs from prior autonomous development sessions. Solutions are reusable; specs are per-feature. +## Roadmap (durable — read FIRST before planning any milestone) + +- [v1.0 roadmap](ROADMAP.md) — M1→M7 dependency graph, 5 hard rails, 10 validation constraints, target package layout, language + scanner coverage. If in-conversation scope disagrees with this file, this file wins. + ## Solutions (architecture patterns + conventions) - [SCIP replaces LSP for code-graph oracle edges](solutions/architecture-patterns/scip-replaces-lsp.md) — one-shot indexers beat stateful LSP clients for compiler-grade graph edges. diff --git a/.erpaval/ROADMAP.md b/.erpaval/ROADMAP.md new file mode 100644 index 00000000..41a4b5a9 --- /dev/null +++ b/.erpaval/ROADMAP.md @@ -0,0 +1,219 @@ +# OpenCodeHub v1.0 Roadmap + +**Source**: `https://dw5vh8cb4iz6i.cloudfront.net/artifacts/och-roadmap/opencodehub-roadmap-2026-05-05.html` (CloudFront signed URL, expires 2026-05-05). +**Extracted**: 2026-05-05. +**Owner**: Laith Al-Saadoon (sole user — rip-and-replace latitude). + +This is the durable roadmap reference. If it conflicts with in-conversation scope, this file wins. Durable by design — committed to survive context compaction. + +## Product thesis + +OpenCodeHub is a personal, local-first, self-hosted OSS code-intelligence hub exposing deterministic cross-repo symbol graphs and SARIF findings through stdio MCP and CLI only. Two-surface product per brainstorm 013: + +- **Surface 1 — laptop artifact factory (P0)**: Claude Code plugin over stdio MCP. `codehub-document`, `codehub-pr-description`, `codehub-onboarding`, `codehub-contract-map`. Visible, immediate wedge. +- **Surface 2 — CI action surface (P1, deferred)**: OSS GH Actions + GitLab templates shelling `codehub` CLI. Structural, slower wedge. Waits on surface-1 adoption. + +## Five hard rails (non-negotiable) + +1. Self-hosted OSS only — no hosted / managed / SaaS / OCH-operated tier. +2. Stdio MCP only — no remote / HTTP MCP. +3. No agent SDK — no Python / TS / claude-hooks / framework adapters. +4. No LLM in query path — index-time summarizer is the sole exception (persisted, citation-validated, opt-in `--llm`). +5. No web UI / eval-server / IDE plugin / LSP / model fine-tuning. + +## Milestone dependency graph + +``` +M1 → M2 → (M3 ∥ M4) → (M5 ∥ M6) → M7 +``` + +Sequenced by dependency only. No calendar estimates. + +## M1 — Stabilize (COMPLETE) + +14 commits on `feat/v1-m1-m2`, landed via PR #53 squash-merge `4431b53`. PASS-WITH-CONCERNS. + +| Task | Scope | Commits | +|------|-------|---------| +| T-M1-1 | Dirty-tree guard on analyze fast-path | `d3fa11b`, `b5e7068`, `fcdd9c9` | +| T-M1-2 | Real incremental via `loadPreviousGraph` snapshot; graphHash byte-identity preserved | `7b100fd`, `cca3c34`, `7ebe4eb` | +| T-M1-3 | `EmbeddingHashCacheAdapter` 3-tier content-hash skip; `--force` re-embeds | `3cfb0cf`, `cca3c34`, `8576f53` | +| T-M1-4 | SARIF symbol-level `FOUND_IN` edges via enclosing-symbol lookup | (in T-M1-2 block) | +| T-M1-5 | Delete 5 canned MCP prompts; skills replace | `73d1375`, `b95cc90`, `a6a210f` | + +**Open concerns** (non-blocking): +- **C1**: `stringArrayField []→NULL` round-trip asymmetry at `analyze.ts:722-730` + `duckdb-adapter.ts:1353-1359` can drift `canonicalJson` hashes. Tracked, pre-M3 cleanup. + +## M2 — Repo split + package surgery (COMPLETE) + +14 commits on `feat/v1-m1-m2`, landed via PR #53. + +| Task | Scope | Commits | +|------|-------|---------| +| T-M2-1 | Extract `packages/eval` + `packages/gym` + `bench/` → `opencodehub-testbed` repo | `53d9b88`, `f6f5f68`, `6d5bc2c` | +| T-M2-2 | Remove `codehub eval-server` HTTP surface | `60b2982`, `1a1ff05` | +| T-M2-3 | Remove `packages/docs` Starlight + `pages.yml`; retain `docs/adr/` | `690ca5e`, `d95df3c` | +| T-M2-4 | `@opencodehub/policy` v1 (3 rule types: `blast_radius_max`, `license_allowlist`, `ownership_required`); wire into `verdict` | `f25b196`, `9890e17`, `d8bfd15`, `4732396` | +| T-M2-5 | Extract `@opencodehub/wiki` workspace package; compat shim in analysis | `6fcc2f0`, `c538f2d`, `dd624ca` | + +## M3 — LadybugDB phase-1 (PENDING, parallel with M4) + +Replace recursive-CTE traversals with polymorphic rel-table-per-edge schema (**corrected 2026-05-05** — the v1 roadmap proposed a single rel-table with a `type` column; LadybugDB docs recommend one named rel table per edge kind with multiple `FROM/TO` pairs for columnar predicate pushdown). Current OCH edge-kind count is **23** (post-M2 additions `FOUND_IN`, `DEPENDS_ON`, `OWNED_BY`, `WRAPS`, `QUERIES`, `REFERENCES`, `ACCESSES`), not 21 as originally estimated. + +LadybugDB = community successor to Kuzu (Apple acquisition). Pre-1.0 with ABI breaks every few months. **Current npm package: `@ladybugdb/core@0.16.1`** (released 2026-05-04, one day before roadmap review). GitNexus pins 0.15.2. Source-level naming uses `GraphDbStore` / `graphdb-adapter.ts` / `graphdb-pool.ts` to stay within `scripts/check-banned-strings.sh` limits — the `ladybug` and `kuzu` literals are rejected in tracked source files; the `@ladybugdb/core` dep in `package.json` is permitted under package-scope precedent. + +| Task | Scope | Dependency | Test gate | +|------|-------|-----------|-----------| +| T-M3-1 | Implement `LbugStore` behind `IGraphStore` seam, gated by `CODEHUB_STORE=lbug` | M2 | graphHash parity suite | +| T-M3-2 | Pool-adapter lifted from GitNexus `pool-adapter.ts` (612 LOC); LadybugDB `.query()` segfaults on concurrent calls | M3-1 | Concurrent query test | +| T-M3-3 | Single `CodeRelation` rel-table + per-kind DDL replaces ~60-column polymorphic nodes table | M3-2 | MATCH pattern tests | +| T-M3-4 | graphHash parity test suite — advance iff `DuckStore.graphHash === LbugStore.graphHash` on corpus | M3-3 | CI gate: byte-identical hash | +| T-M3-5 | Convert `sql` MCP tool output to `cypher` (dual-emit during phase 1, drop `sql` at M7) | M3-4 | MCP tool signature tests | +| T-M3-6 | ADR documenting swap rationale + 3-phase plan | M3-5 | Documentation reviewed | + +**Fallbacks**: DuckDB remains legacy through M7. Apache AGE on Postgres 18 is survivability fallback if LadybugDB breaks beyond repair (documented, not implemented until M7). + +## M4 — Language expansion (PENDING, parallel with M3) + +| Task | Scope | Notes | +|------|-------|-------| +| T-M4-1 | `scip-clang` adapter | Needs `compile_commands.json`, 2 GB RAM/core guard | +| T-M4-2 | `scip-ruby` adapter | Sorbet install workflow | +| T-M4-3 | `scip-dotnet` adapter | — | +| T-M4-4 | Kotlin promotion (distinct from Java) | `scip-kotlin` v0.6.0 via `scip-java` | +| T-M4-5 | COBOL regex hot path | ~1 ms/file; `copybook`, `CICS`, `PARAGRAPH`, `PERFORM` extraction | +| T-M4-6 | COBOL ProLeap v4.0.0 backend | ANTLR4/JVM Java subprocess, `--allow-build-scripts` gated. tree-sitter-cobol (v0.1.1, 2023-02-01 — no newer tagged release) remains unreliable. **ProLeap is NOT published to Maven Central** (`search.maven.org` returns 0; last GitHub Release v2.4.0 from 2018); M4-6 must `git clone + mvn install` OR ship a prebuilt JAR under `vendor/proleap/`. ProLeap does not ship a CLI — need a small Java `main` wrapper. | +| T-M4-7 | Framework detection 5-stage pipeline | New `@opencodehub/frameworks` package. No OSS drop-in; custom curated-registry. | + +**Framework detection stages** (each emits `{framework, version?, confidence, evidence[]}`): +1. Manifest presence (`package.json`, `pyproject.toml`, `pom.xml`, `Gemfile`, `go.mod`, `Cargo.toml`) +2. Lockfile + exact versions (semver-aware, curated registry) +3. Config AST (`astro.config.mjs`, `next.config.js`, `vite.config.ts`, `spring.factories`) +4. Folder convention (`app/`, `pages/`, `src/main/java/`, `config/routes.rb`) +5. Import / SCIP usage patterns (`import fastapi`, `from django.db`, `@SpringBootApplication`) + +## M5 — Deterministic code-packs (PENDING, parallel with M6) + +Depends on M4. + +| Task | Scope | +|------|-------| +| T-M5-1 | `@opencodehub/pack` package with 9-item BOM contract | +| T-M5-2 | PageRank extraction from `scip-ingest/materialize.ts` dead code → `analysis/page-rank.ts` | +| T-M5-3 | `codehub code-pack` CLI subcommand + MCP tool | +| T-M5-4 | Byte-identity determinism test suite | +| T-M5-5 | `codehub-code-pack` SKILL.md | + +**9-item code-pack BOM** (byte-identical given same commit, tokenizer, budget): +1. `manifest.json` — pack_hash, commit SHA, tokenizer ID, schema version, counts +2. PageRank-ranked symbol skeleton +3. File tree with framework labels +4. Dependency graph / lockfile slice (exact versions) +5. Top-N AST-chunked files with byte offsets +6. SCIP-grounded cross-refs (community clusters + call graph) +7. Optional embeddings sidecar (`.parquet`) +8. Salient docstrings / SARIF findings by severity + rule +9. LICENSES / NOTICES + README.md + full determinism contract + +## M6 — Cross-repo federation (PENDING, parallel with M5) + +Depends on M5. + +| Task | Scope | +|------|-------| +| T-M6-1 | First-class `Repo` entity in graph | +| T-M6-2 | `group_list`, `group_status`, `group_contracts`, `group_query` MCP tools | +| T-M6-3 | `codehub-contract-map` skill (group-only, Mermaid consumer → producer) | +| T-M6-4 | Cross-repo link graph in `codehub-document --group` | +| T-M6-5 | `AMBIGUOUS_REPO` sentinel when ≥ 2 repos indexed without explicit `repo:` | + +## M7 — LadybugDB default, DuckDB legacy (PENDING) + +Depends on M3 + M6. + +| Task | Scope | +|------|-------| +| T-M7-1 | Flip default backend to `CODEHUB_STORE=lbug` | +| T-M7-2 | Retain DuckDB only for temporal analytics | +| T-M7-3 | Drop dual-emit `sql|cypher` → `cypher`-only | +| T-M7-4 | Final graphHash parity audit across testbed corpus | +| T-M7-5 | Apache AGE / Postgres 18 escape hatch documented (not implemented) | + +## Target package layout at end of roadmap + +**Core (11 packages, ~400 files from ~970)**: +- `@opencodehub/cli` — `codehub` binary, 22+ subcommands (adds `verdict`, `code-pack`) +- `@opencodehub/mcp` — stdio MCP (29+ tools, 0 prompts) +- `@opencodehub/analysis` — request-time queries (PageRank, blast, impact) +- `@opencodehub/ingestion` — scan + materialize pipeline +- `@opencodehub/scip-ingest` — SCIP proto parsing +- `@opencodehub/storage` — `IGraphStore` + `DuckStore` + `LbugStore` +- `@opencodehub/embed` (née embedder) — transformers.js default + HTTP endpoint +- `@opencodehub/summarizer` — Bedrock Haiku 4.5, index-time only +- `@opencodehub/sarif` — SARIF 2.1.0 schemas + baseline diff +- `@opencodehub/scanners` — 20-scanner orchestrator +- `@opencodehub/core-types` — shared types + +**New (4 packages)**: +- `@opencodehub/frameworks` — 5-stage framework detection +- `@opencodehub/pack` — deterministic code-pack generator +- `@opencodehub/policy` — `opencodehub.policy.yaml` + evaluator (M2 shipped) +- `@opencodehub/wiki` — deterministic wiki (M2 shipped) + +## Language coverage targets at v1.0 + +| Language | Tree-sitter | SCIP | Frameworks | Status | +|----------|-------------|------|-----------|--------| +| TypeScript / JavaScript | ✅ | scip-typescript 0.4.0 | Next.js, Nest, Astro, Remix, Vite, Express | Active | +| Python | ✅ | scip-python | FastAPI, Django, Flask, LangChain, Pydantic | Active | +| Go | ✅ | scip-go 0.2.4 | stdlib, Gin, Echo | Active | +| Java | ✅ | scip-java 0.12.3 | Spring Boot, Micronaut, Gradle, Maven | Active | +| Scala | ✅ | scip-java 0.12.3 | Play, Akka | Active (via java) | +| Kotlin | ✅ | scip-kotlin 0.6.0 | Ktor, Android | M4 promotion | +| Ruby | ✅ | scip-ruby 0.4.7 | Rails, Sinatra | M4 | +| C / C++ | ✅ | scip-clang 0.4.0 | CMake, Conan | M4 | +| C# / .NET | ✅ | scip-dotnet | ASP.NET, EF Core | M4 | +| Rust | ✅ | Gap | cargo, Axum, Tokio | Tree-sitter only; SCIP blocked | +| Swift | ✅ | Gap | SwiftUI, Vapor | Tree-sitter only | +| COBOL | ❌ | None | CICS, IMS, JCL | Regex hot path + ProLeap v4 (gated) | + +## Scanner pipeline (20 scanners at v1.0) + +SARIF 2.1.0 ingestion + baseline diff + `codehub verdict` CI exit codes + `ci-init` workflow generation. + +- **SAST**: Semgrep, CodeQL, Bandit (Py), Brakeman (Rb), GoSec, detect-secrets +- **SCA / license**: OSV-Scanner, internal `license_audit`, CycloneDX/SBOM +- **Type**: tsc, pyright, mypy, ruff-type +- **Lint**: Biome, ruff, golangci-lint, clippy +- **Fingerprinting**: `opencodehub/v1` via `{rule_id, symbol_id, hash(snippet)}` for stable baseline diff across formatters + +## Validation constraints (every milestone must satisfy all 10) + +| # | Constraint | Check | +|---|-----------|-------| +| 1 | Stdio MCP + CLI only; no HTTP surfaces | `rg -n 'express\|fastify\|http.createServer' packages/ → 0` | +| 2 | No LLM in query path | No `@aws-sdk/client-bedrock-runtime` outside `packages/summarizer/` | +| 3 | Narrative / LLM features ship as skills | `plugins/opencodehub/skills/*/SKILL.md` exists per narrative tool | +| 4 | Fixtures / evals / gyms in testbed repo | absent from core post-M2 | +| 5 | `mise run check` exit 0 | per commit | +| 6 | `graphHash` byte-identical full vs incremental | CI gate | +| 7 | Deterministic code-pack | same commit + tokenizer + budget → same bytes | +| 8 | No time estimates | sequenced by dependency graph only | +| 9 | SARIF 2.1.0 conformance | Zod passthrough + sarif-sdk spec tests | +| 10 | 20-scanner pipeline coverage | scanner registry enumerated | + +## Explicitly rejected (no exceptions) + +- Hosted / managed / SaaS tier +- Remote / HTTP MCP server +- Agent SDK (Python, TS, claude-hooks, framework adapters) +- `grounding_pack` MCP compositor +- OpenCodeHub-branded coding agent +- LLM-based PR review +- Hosted review UI (GitHub Checks + PR comments only) +- IDE plugin / LSP +- Model fine-tuning + +## Rip-and-replace latitude + +1 active user. Roadmap explicitly sanctions rip-and-replace where it produces a better shape. No breaking-change budget to preserve beyond the graphHash byte-identity invariant and the MCP tool contract (tools may be renamed/replaced as long as the skill layer is updated in the same change). diff --git a/.erpaval/specs/004-m3-m4/spec.md b/.erpaval/specs/004-m3-m4/spec.md new file mode 100644 index 00000000..aee41f06 --- /dev/null +++ b/.erpaval/specs/004-m3-m4/spec.md @@ -0,0 +1,271 @@ +# EARS Spec 004 — M3 LadybugDB phase-1 + M4 Language expansion + +**Session**: session-a591fa · **Branch**: `feat/v1-m3-m4` · **Parent roadmap**: `.erpaval/ROADMAP.md` §M3 + §M4 + +## Context (Explore + Research consolidated) + +### M3 — LadybugDB phase-1 + +- `IGraphStore` seam at `packages/storage/src/interface.ts:11-64` is already the abstraction point. No shape change needed. +- `graphHash` is computed in `packages/core-types/src/graph-hash.ts:20-45` from the **in-memory `KnowledgeGraph`**, never from store rows. Parity test: `graph → LbugStore → rebuildGraphFromStore → graphHash === original`. Template exists at `packages/storage/src/duckdb-adapter.test.ts:89,206-229`. +- **Current edge-kind count is 23** (`duckdb-adapter.ts:71-96`) — roadmap's "21 types" is stale; OCH has drifted past with `FOUND_IN`, `DEPENDS_ON`, `OWNED_BY`, `WRAPS`, `QUERIES`, `REFERENCES`, `ACCESSES`. OCH uses `PROCESS_STEP` where GitNexus uses `STEP_IN_PROCESS` (banned literal). +- **LadybugDB pattern correction** (supersedes roadmap L58): idiomatic LadybugDB uses **polymorphic rel tables — one named rel table per edge kind, each with multiple `FROM/TO` pairs**. NOT a single `CodeRelation` rel table with a `type` property column — that defeats columnar predicate pushdown. Research URL: `docs.ladybugdb.com/cypher/data-definition/create-table`. +- **npm package**: `@ladybugdb/core@^0.16.1` (latest as of 2026-05-04). GitNexus pins 0.15.2. `lbug@0.14.3` is a stale mirror — ignore. +- **Concurrency**: one process-wide `READ_WRITE` `Database` + pool of `Connection` objects. GitNexus's `pool-adapter.ts` (611 LOC) is user-space wrapper, not library convention — worth lifting but re-audit for current (v0.16) behavior vs v0.15. +- **Banned literals**: `kuzu`, `ladybug`, `STEP_IN_PROCESS`, `duckpgq` are banned in tracked source by `scripts/check-banned-strings.sh`. `@ladybugdb/core` in `package.json` is allowed (not a banned form). `.erpaval/` is excluded from the scan. The `LbugStore` class name and file paths `lbug-adapter.ts` / `lbug-pool.ts` use the "lbug" token which triggers the banned literal. **Resolution**: rename everything to `GraphDbStore` / `graphdb-adapter.ts` / `graphdb-pool.ts` at the source level; keep `@ladybugdb/core` as the dep name (the package scope is exempt by precedent). + +### M4 — Language expansion + COBOL + framework detection + +- 5 live SCIP adapters in `packages/scip-ingest/src/runners/index.ts:18` as a string union `"typescript" | "python" | "go" | "rust" | "java"`. No provider-registry abstraction. Adding `clang | ruby | dotnet | kotlin` = extend union + add `buildCommand` cases. +- **No scip-* binary downloads**: `codehub setup` only handles embeddings weights + plugin. New adapters assume binaries on `$PATH` (returns `kind: "missing"` on ENOENT). M4 must add `scip-downloader.ts` mirroring `embedder-downloader.ts` (sha256 pin + atomic rename). +- 15 tree-sitter grammars in `grammar-registry.ts:36-52`, compile-time-enforced via `satisfies` on `LanguageId`. **No regex-provider escape hatch**; COBOL T-M4-5 cannot reuse the registry without introducing one. +- 23-framework catalog at `frameworks-catalog.ts:437`, inline in `packages/ingestion`. Emits `{name, category, confidence: "deterministic"|"heuristic"|"composite", signals[], variant?, version?, parentName?}` — roadmap asks for numeric `confidence` + `evidence[]`. Plan must choose: **keep current discriminator** (string tag) + rename `signals` → `evidence` (cheaper), or go numeric (bigger change, arguable utility for 1 user). +- **5 detection stages coverage**: manifest ✅, lockfile ❌ (ignored today), config-AST ❌ (exact-match only, no parse), folder-convention partial, import/SCIP ❌. +- **No JVM subprocess prior art** — ProLeap v4 (T-M4-6) is greenfield. Grep empty for `java -jar`, `spawn.*java`, `jbang`. Needs new package + JRE probe. +- **ProLeap NOT on Maven Central** — `search.maven.org` returns `numFound: 0` for `io.github.uwol:proleap-cobol-parser`; latest GitHub Release is v2.4.0 (2018). M4-6 must `git clone + mvn install` into a vendored JAR OR ship a prebuilt JAR under `vendor/proleap/`. +- **tree-sitter-cobol published releases dead** (last tagged v0.1.1, 2023-02-01 per GitHub Releases API). Commit activity on default branch through 2025 but no tagged release. COBOL strategy stays as roadmap spec'd: regex hot path primary + ProLeap deep-parse gated. +- **`--allow-build-scripts`** is internal `RunIndexerOptions` boolean at `runners/index.ts:25` — never surfaced at CLI. T-M4-6 needs CLI flag + plumbing. + +### Banned-string sensitivities + +- `kuzu`, `ladybug`, `STEP_IN_PROCESS` are guardrail-banned in tracked source. +- Source-level naming: `GraphDbStore` / `graphdb-adapter.ts` / `graphdb-pool.ts` (not `LbugStore`). +- `@ladybugdb/core` in `package.json` — precedent: `@opencodehub/*` scoped packages with banned substrings are allowed when the scope identifier is the whole token. Verify by running `bash scripts/check-banned-strings.sh` after adding the dep; if it flags, add an allowlist exclusion for `package.json` + `pnpm-lock.yaml` (already excluded). + +## Ubiquitous requirements + +- **U1**: The v1.0 roadmap's graphHash byte-identity invariant MUST hold across both stores — `graph → DuckDbStore → rebuildGraphFromStore → graphHash` and `graph → GraphDbStore → rebuildGraphFromStore → graphHash` MUST be equal. +- **U2**: No tracked source file MUST introduce the banned literals `kuzu`, `ladybug`, `STEP_IN_PROCESS`, `heuristicLabel`, `codeprobe`, or `STEP_IN_FLOW`. `bash scripts/check-banned-strings.sh` MUST exit 0 post-commit. +- **U3**: `mise run check` MUST exit 0 after every commit. +- **U4**: Every new package MUST carry `@opencodehub/` naming, Apache-2.0 license, `type: module`, `tsc --noEmit` clean. +- **U5**: No LLM calls in any M3/M4 path outside the existing `@opencodehub/summarizer` package. + +## M3 — Event-driven requirements + +- **E-M3-1**: When `CODEHUB_STORE=lbug` is set, `analyze`, `query`, `context`, `impact`, and `sql` CLI/MCP surfaces MUST route through `GraphDbStore` instead of `DuckDbStore`. +- **E-M3-2**: When the `sql` MCP tool receives a `cypher` input field, it MUST evaluate as read-only Cypher against `GraphDbStore`. Write operations (`CREATE`, `DELETE`, `SET`, `MERGE`) MUST be rejected by `cypher-guard.ts` (mirror of `sql-guard.ts`). +- **E-M3-3**: When both `sql` and `cypher` inputs are provided to the `sql` MCP tool, the tool MUST reject the call with a clear "choose one" message. + +## M3 — State-driven requirements + +- **S-M3-1**: While `CODEHUB_STORE` is unset or `=duck`, `DuckDbStore` remains the default; `GraphDbStore` is not loaded. +- **S-M3-2**: While `@ladybugdb/core` is absent (unreachable import — should not happen because it's a hard dep, but CI platforms without prebuilt binaries will surface this), `GraphDbStore.open()` MUST fail with a clear "`@ladybugdb/core` native binding unavailable on this platform; use `CODEHUB_STORE=duck`" message — not a bare module-not-found stack trace. +- **S-M3-3**: While a `GraphDbStore` database file exists from a prior `@ladybugdb/core` version (ABI mismatch), `open()` MUST emit a runbook hint pointing at the re-analyze path (`codehub analyze --force`), not silently truncate. + +## M3 — Unwanted-behavior requirements + +- **W-M3-1**: `GraphDbStore` MUST NOT call `conn.query()` concurrently against a single `Connection` — the pool adapter enforces one-query-per-connection at a time. +- **W-M3-2**: Cypher write operations (`CREATE`, `DELETE`, `SET`, `MERGE`, `REMOVE`) MUST NOT pass the `cypher-guard.ts` read-only check. The `sql` MCP tool stays read-only regardless of store backend. +- **W-M3-3**: The M3 phase-1 MUST NOT flip the default backend to `lbug`. That is T-M7-1. + +## M3 — Acceptance criteria + +### AC-M3-1: GraphDbStore scaffolding + +- [ ] `packages/storage/src/graphdb-adapter.ts` — `GraphDbStore implements IGraphStore`, constructor takes path, lazy-imports `@ladybugdb/core` +- [ ] `packages/storage/src/graphdb-schema.ts` — DDL translator; per-kind `CREATE NODE TABLE` + one polymorphic rel table per edge kind +- [ ] `packages/storage/src/graphdb-pool.ts` — lifted from GitNexus `pool-adapter.ts` (611 LOC), renamed, internals audited for v0.16 API compatibility +- [ ] `packages/storage/src/index.ts` — export `GraphDbStore`; add `openStore(opts)` factory reading `CODEHUB_STORE` +- [ ] `packages/storage/package.json` — add `@ladybugdb/core: ^0.16.1` as hard dep (direct dependency, not optional peer — user-approved 2026-05-05) +- [ ] Banned-strings gate passes (no `kuzu`/`ladybug` in source) +- [P] +- **Dependencies**: none + +### AC-M3-2: Pool adapter + concurrency tests + +- [ ] `graphdb-pool.ts` integration test: 100 concurrent reads against one Database do not segfault or deadlock +- [ ] Checkout/checkin queue semantics preserved from GitNexus pool (`MAX_CONNS_PER_REPO=8`, 15s waiter timeout, 30s query timeout, 60s idle sweep) +- [ ] Timeout propagates into `IGraphStore.query()` `timeoutMs` correctly +- **Dependencies**: AC-M3-1 + +### AC-M3-3: Schema translation + round-trip + +- [ ] All 23 edge kinds from `duckdb-adapter.ts:71-96` have corresponding rel tables in `graphdb-schema.ts` +- [ ] `PROCESS_STEP` (OCH-native, not the banned `STEP_IN_PROCESS`) maps to a rel table named `ProcessStep` (or similar — no banned literal) +- [ ] `bulkLoad(graph, "replace")` + `rebuildGraphFromStore(graphdbStore)` round-trip produces a graph with identical nodes, edges, and properties as the input +- **Dependencies**: AC-M3-1 + +### AC-M3-4: graphHash parity gate (CI) + +- [ ] New file `packages/storage/src/graph-hash-parity.test.ts` +- [ ] Against 3 fixture graphs (small, medium, large) assert `duckHash === graphdbHash` +- [ ] Wired into `mise run check` +- [ ] Test runs in <30s so it stays in the hot validate path +- **Dependencies**: AC-M3-3 + +### AC-M3-5: sql MCP tool dual-emit (sql | cypher) + +- [ ] `packages/mcp/src/tools/sql.ts` accepts optional `cypher` input field +- [ ] `packages/storage/src/cypher-guard.ts` mirrors `sql-guard.ts` — allows `MATCH`, `RETURN`, `WITH`, `WHERE`, `ORDER BY`, `LIMIT`, `SKIP`, `UNWIND`, `CALL READ_ONLY_PROCEDURES`; rejects writes +- [ ] When `CODEHUB_STORE=duck`, `cypher` input returns "cypher unavailable without `CODEHUB_STORE=lbug`" +- [ ] Timeout path shared between sql + cypher branches +- **Dependencies**: AC-M3-4 + +### AC-M3-6: ADR — LadybugDB swap rationale + +- [ ] `docs/adr/NNNN-ladybugdb-graph-store.md` (numeric pick from existing ADR numbering) +- [ ] Documents the 3-phase plan (M3 opt-in → M7 default → DuckDB legacy-only), polymorphic rel-table-per-kind decision, pool adapter rationale, banned-literal renaming strategy, Apache AGE fallback +- [ ] Does NOT contain banned literals outside the banned-strings allowlist scope +- **Dependencies**: AC-M3-5 + +## M4 — Event-driven requirements + +- **E-M4-1**: When `codehub analyze` runs on a repo containing `*.c`/`*.cpp`/`*.h`, it MUST invoke `scip-clang` if the binary is on `$PATH` or was installed via `codehub setup --scip=clang`. +- **E-M4-2**: When the user invokes `codehub setup --scip=`, the CLI MUST download the platform-specific binary, verify its sha256 against the pinned hash, and install into `~/.codehub/bin/` (or equivalent). +- **E-M4-3**: When `codehub analyze` encounters COBOL files (`.cbl`, `.cob`, `.cpy`), it MUST run the regex hot path (T-M4-5) unconditionally, and MUST run the ProLeap deep-parse (T-M4-6) only when `--allow-build-scripts=proleap` is passed. +- **E-M4-4**: When the 5-stage framework-detection pipeline emits a detection, the result MUST include `{name, version?, confidence, evidence[]}` where `confidence` is one of the discriminator strings (`"deterministic"|"heuristic"|"composite"`) AND `evidence[]` lists the stage(s) that produced the signal. + +## M4 — State-driven requirements + +- **S-M4-1**: While a SCIP adapter's binary is not installed, `codehub analyze` MUST skip that language cleanly (not crash) and emit a setup hint. +- **S-M4-2**: While `java --version` fails or reports < 17, `codehub analyze --allow-build-scripts=proleap` MUST refuse to run and emit a clear install hint for JRE 17+. +- **S-M4-3**: While the ProLeap JAR is not vendored under `vendor/proleap/proleap-cobol-parser-.jar`, `codehub analyze --allow-build-scripts=proleap` MUST fail with the specific missing-jar path. + +## M4 — Unwanted-behavior requirements + +- **W-M4-1**: The COBOL ProLeap path MUST NOT run by default — only when the user explicitly passes `--allow-build-scripts=proleap`. This protects against unexpected JVM subprocess spawns. +- **W-M4-2**: The 5-stage framework-detection pipeline MUST NOT call out to network / LLM / any service. It's a pure-local file-system + AST inspection. +- **W-M4-3**: Scip adapters MUST NOT download binaries at analyze time. All downloads happen via `codehub setup`. +- **W-M4-4**: The framework-catalog MUST NOT double-trigger when both manifest and lockfile signals fire (the composite already handles this — do not regress). + +## M4 — Acceptance criteria + +### AC-M4-1: scip-clang adapter + +- [ ] Add `"clang"` to `IndexerKind` union in `packages/scip-ingest/src/runners/index.ts` +- [ ] `buildCommand("clang", opts)` → `scip-clang index --output ` from project root with `compile_commands.json` preflight check +- [ ] `scip-clang` version pin: v0.4.0 (2026-02-23), binary URL pattern `github.com/sourcegraph/scip-clang/releases/download/v0.4.0/scip-clang-x86_64-{linux|darwin}` +- [ ] Tests: mock-binary invocation, missing-binary skip path, `compile_commands.json` missing → specific error +- [P] +- **Dependencies**: AC-M4-0 (downloader — see below) + +### AC-M4-2: scip-ruby adapter + +- [ ] Add `"ruby"` to `IndexerKind` union +- [ ] `buildCommand("ruby")` → `scip-ruby --index-file ` (verify invocation against scip-ruby v0.4.7 docs) +- [ ] Pin: v0.4.7 (2024-11-07), multi-arch: linux-x64, linux-arm64, darwin-x64, darwin-arm64 +- [P] +- **Dependencies**: AC-M4-0 + +### AC-M4-3: scip-dotnet adapter + +- [ ] Add `"dotnet"` to `IndexerKind` union +- [ ] `buildCommand("dotnet")` → `scip-dotnet index -o ` with .NET SDK 8+ probe (exits with install hint if missing) +- [ ] Pin: v0.2.12; installed via `dotnet tool install --global scip-dotnet` OR vendored +- [P] +- **Dependencies**: AC-M4-0 + +### AC-M4-4: scip-kotlin adapter (promotion from tree-sitter only) + +- [ ] Add `"kotlin"` to `IndexerKind` union +- [ ] `buildCommand("kotlin")` — confirm invocation pattern against scip-kotlin v0.6.0 docs (standalone, NOT bundled in scip-java) +- [ ] Tests differentiate Kotlin from Java in `detectLanguages()` (Kotlin must now produce its own SCIP, not ride on Java) +- [P] +- **Dependencies**: AC-M4-0 + +### AC-M4-0: codehub setup --scip= downloader + +- [ ] New file `packages/cli/src/scip-downloader.ts` — mirror of `embedder-downloader.ts` +- [ ] Platform detection: linux-x64, linux-arm64, darwin-x64, darwin-arm64 (windows out of scope for v1) +- [ ] sha256-pinned downloads, atomic rename, idempotent re-run +- [ ] Subcommand: `codehub setup --scip=` or `codehub setup --scip=all` +- [ ] Tests: pinned-hash verification, pin-mismatch refusal, concurrent setup guard +- **Dependencies**: none (blocks AC-M4-1..4) + +### AC-M4-5: COBOL regex hot path + +- [ ] New file `packages/ingestion/src/parse/cobol-regex.ts` +- [ ] Extracts `copybook`, `CICS`, `PARAGRAPH`, `PERFORM` identifiers from `.cbl`, `.cob`, `.cpy` files; ≤1ms per file on 1000-line fixture +- [ ] Emits `CodeElement` nodes with confidence `"heuristic"` +- [ ] Wired into the parse pipeline as a new regex-provider escape hatch: extends `LanguageId` union to include `"cobol"` with a regex-provider discriminator +- [ ] Tests: NIST COBOL85 test fixtures from ProLeap's test corpus +- [P] +- **Dependencies**: none + +### AC-M4-6: COBOL ProLeap deep-parse + +- [ ] New package `packages/cobol-proleap/` — `@opencodehub/cobol-proleap`; `index.ts` + JVM subprocess wrapper +- [ ] Loads JAR from `~/.codehub/vendor/proleap/proleap-cobol-parser-.jar` (not committed; fetched on-demand — user-approved 2026-05-05) +- [ ] `codehub setup --cobol-proleap` subcommand downloads + sha256-verifies + installs the prebuilt JAR (mirrors `scip-downloader.ts` shape) +- [ ] Builds small Java `main` wrapper (`cobol_to_scip.java` — maps ProLeap ASG to SCIP-compatible JSON) since ProLeap doesn't ship a CLI. The wrapper itself is committed under `packages/cobol-proleap/java/`; ProLeap JAR stays on-demand. +- [ ] Gated by `--allow-build-scripts=proleap` CLI flag (new surface); unset → regex hot path only +- [ ] Amortizes JVM startup by batching files per invocation +- [ ] Tests: synthetic COBOL file round-trip, JAR-missing failure, JRE-missing failure, graceful fallback to regex hot path on ProLeap crash +- [ ] `commitlint.config.mjs` — add `cobol-proleap` to scope-enum in the first commit +- **Dependencies**: AC-M4-5 (fallback path) + AC-M4-0 (downloader) + +### AC-M4-7: @opencodehub/frameworks extraction + 5-stage pipeline + +- [ ] New package `packages/frameworks/` — moves `framework-detector.ts`, `frameworks-catalog.ts`, `frameworks.ts`, `manifests.ts` out of `packages/ingestion/src/pipeline/profile-detectors/` +- [ ] Stage 2 (lockfile): parse `package-lock.json`, `pnpm-lock.yaml`, `Gemfile.lock`, `poetry.lock`, `uv.lock`, `Cargo.lock` for exact versions +- [ ] Stage 3 (config-AST): add `next.config.{js,mjs,ts}`, `astro.config.mjs`, `vite.config.*` AST parse via existing tree-sitter or regex-pragmatic matchers (no new deps) +- [ ] Stage 5 (import/SCIP): consume the graph's `IMPORTS` edges — if any SCIP-resolved symbol targets a registered framework's root module (e.g., `fastapi`, `django.db`), emit a detection +- [ ] Re-export from `packages/ingestion` for backward compat +- [ ] `FrameworkDetection` shape: rename `signals` → `evidence`; keep discriminator `confidence` +- [ ] `commitlint.config.mjs` — add `frameworks` to scope-enum in the first commit +- [P] +- **Dependencies**: none + +### AC-M4-8: Validate + PR + +- [ ] `mise run check` exits 0 post-merge +- [ ] `graphHash` byte-identity test still passes (M3 parity + M4 additions) +- [ ] `bash scripts/check-banned-strings.sh` exits 0 +- [ ] New tests bring totals to ~1,700+ (from current 1,449) +- [ ] PR `feat/v1-m3-m4 → main` opened with structured body listing each AC + commit ranges +- **Dependencies**: AC-M3-6, AC-M4-6, AC-M4-7 (terminal) + +## Architectural decisions + +1. **Rel-table-per-edge, not single `type` column.** Supersedes roadmap wording. Rationale: columnar predicate pushdown, no full-scan filter, matches LadybugDB idiom documented in `docs.ladybugdb.com/cypher/data-definition/create-table`. +2. **Store names do NOT use the `Lbug` or `Ladybug` prefix in source.** `GraphDbStore` / `graphdb-adapter.ts` / `graphdb-pool.ts` — passes the banned-strings guardrail cleanly. Package dep stays `@ladybugdb/core` (package-scope identifiers are precedent-allowed). +3. **`sql` MCP tool keeps its name; adds optional `cypher` input.** Not a new tool. No MCP tool-count bump yet (stays at 28 live + 5 deleted prompts = 28 tools surface). M7 will rename to `graph_query` and drop the sql branch. +4. **COBOL regex hot path first; ProLeap is gated deep-parse.** Roadmap sequenced correctly — regex provides the 80% coverage at ~1ms/file; ProLeap adds AST precision for users who opt in via `--allow-build-scripts=proleap` and accept the JVM subprocess cost. +5. **`@opencodehub/frameworks` extraction in-milestone.** Roadmap calls for it; AC-M4-7 does both the extraction and the stage-2/3/5 gap fill together — one change, one breaking import for `packages/ingestion`, easier to reason about than staging. +6. **scip-* downloader is AC-M4-0 (prerequisite).** Blocks M4-1..4. Ships as an independent commit. + +## Anti-goals + +- Do NOT change the MCP tool count rhetoric in `CLAUDE.md` or `README.md` — they say "28 tools" and stay at 28 through M3 (no new tools; `sql` gains an input field). +- Do NOT introduce banned literals in tracked source under any milestone. +- Do NOT flip the default `CODEHUB_STORE` backend in M3; that is M7. +- Do NOT vendor a ProLeap JAR over 20 MB without documenting size + license impact in the ADR. +- Do NOT bundle `@ladybugdb/core` as a required dep — it's optional to keep `pnpm install` flicker-free on platforms without the native binary. +- Do NOT call out to the network or spawn LLM calls in M4-7 framework detection — stage-5 uses the existing graph only. +- Do NOT batch M3 + M4 into a single atomic commit; they're independent and parallelizable. Ship per-AC commits. +- Do NOT skip the `scripts/check-banned-strings.sh` gate — every commit runs it via pre-commit hook. + +## Commit protocol (roll-up across all M3 + M4 tasks) + +- Smallest useful commits. Per-AC atomic commits preferred; multi-file ACs split per-file where possible. +- Each commit runs `bash scripts/check-banned-strings.sh` + `pnpm exec biome check --write ` + `pnpm --filter exec tsc --noEmit` + `pnpm --filter test`. +- Every AC's terminal commit additionally runs `mise run check` before pushing. +- Use `isolation: "worktree"` for every parallel Act subagent (M2 lesson). +- Commit messages follow conventional-commits; scope enum already covers `storage`, `scip-ingest`, `ingestion`, `cli`, `mcp`, `repo`, `docs`, `deps`. New `frameworks` scope needs `commitlint.config.mjs` update at the start of AC-M4-7. + +## Parallel wave structure (Plan derives tasks from this) + +``` +Wave 0 (independent prep, fully parallel): + AC-M4-0 (scip downloader) — blocks M4-1..4 + AC-M4-5 (COBOL regex) — independent + AC-M4-7 (frameworks extraction + stages) — independent + AC-M3-1 (GraphDbStore scaffolding) — blocks M3-2..6 + +Wave 1 (parallel): + AC-M3-2 (pool + concurrency) + AC-M3-3 (schema + round-trip) + AC-M4-1 scip-clang + AC-M4-2 scip-ruby + AC-M4-3 scip-dotnet + AC-M4-4 scip-kotlin + AC-M4-6 ProLeap (depends on AC-M4-5) + +Wave 2 (terminal, sequential within track): + AC-M3-4 (graphHash parity gate) + AC-M3-5 (sql dual-emit) + AC-M3-6 (ADR) + AC-M4-8 (validate + PR) +``` + +Total: **13 ACs** across 2 waves. Expected commit count ~25-30 atomic commits on `feat/v1-m3-m4`. diff --git a/.gitignore b/.gitignore index 1f12c656..079fd481 100644 --- a/.gitignore +++ b/.gitignore @@ -37,3 +37,6 @@ examples/fixtures/**/.codehub/ .claude/settings.local.json .claude/worktrees/ .handoff/ +.gitnexus +.claude/skills/gitnexus/ +.claude/skills/generated/ diff --git a/commitlint.config.mjs b/commitlint.config.mjs index a25533be..c0c837be 100644 --- a/commitlint.config.mjs +++ b/commitlint.config.mjs @@ -33,20 +33,20 @@ export default { [ "analysis", "cli", + "cobol-proleap", "core-types", "embedder", - "gym", + "frameworks", "ingestion", - "lsp-oracle", "mcp", "policy", "sarif", "scanners", + "scip-ingest", "search", "storage", "summarizer", "wiki", - "eval", "plugin", "deps", "ci", diff --git a/docs/adr/0011-graph-db-backend.md b/docs/adr/0011-graph-db-backend.md new file mode 100644 index 00000000..59b129c6 --- /dev/null +++ b/docs/adr/0011-graph-db-backend.md @@ -0,0 +1,306 @@ +# ADR 0011 — Graph-DB backend (LadybugDB phase-1) + +- Status: **Proposed** — 2026-05-05 (flips to **Accepted** on the M3 merge). +- Authors: Laith Al-Saadoon + Claude. +- Branch: `feat/v1-m3-m4`. +- Supersedes nothing. Interacts with ADR 0001 (DuckDB backend stays the + default through M6; this ADR records the opt-in second backend and the + phased plan to flip the default in M7). + +## Context + +OpenCodeHub's storage layer was chosen in ADR 0001 with a relational shape: +DuckDB + the `hnsw_acorn` vector index + `fts` BM25 + recursive CTEs over +a single polymorphic `relations` table keyed by a `type` discriminator +column. That shape has held up for the first two milestones, but the M3 +workload surfaces two specific strains the column-store cannot relieve by +configuration alone. + +1. **Recursive-CTE traversals are slow on deep call graphs.** Multi-hop + `impact` and `context` queries land on the `relations(from_id, to_id, + type, …)` table and fan out via `WITH RECURSIVE … USING KEY`. DuckDB's + `USING KEY` implementation is the right algorithm for this shape, but + every hop pays the cost of a `WHERE type = ?` predicate against a table + that stores all 24 edge kinds intermixed. The planner has no columnar + pushdown to narrow the probe to the one kind we care about — the filter + is evaluated after the join, not before. +2. **The polymorphic `type` column defeats columnar predicate pushdown.** + A single rel table with a `type` discriminator was the right shape when + OCH tracked ~10 edge kinds. The M2 additions (`FOUND_IN`, `DEPENDS_ON`, + `OWNED_BY`, `WRAPS`, `QUERIES`, `REFERENCES`, `ACCESSES`) pushed the + count to the **24 kinds** currently declared in + `packages/storage/src/graphdb-schema.ts`. At that cardinality each query + scans a fraction of rows it never uses, and the planner cannot prune + what it cannot see. + +The clean fix is a graph-native store that speaks Cypher, supports +multiple named relationship tables, and keeps each edge kind in its own +physical layout so the planner prunes at the table level, not the row +level. The M3 scope adds that backend behind the existing `IGraphStore` +seam as an opt-in surface. Flipping the default is explicitly out of M3. + +## Decision + +Add a graph-database backend bound to `@ladybugdb/core` (the community +LadybugDB project — the successor to the pre-1.0 Kuzu codebase after +Kuzu's Apple acquisition; see §Provenance at the end of this ADR) +behind the `IGraphStore` interface at +`packages/storage/src/interface.ts`. The store is selected at runtime by +the environment variable `CODEHUB_STORE`: + +- `CODEHUB_STORE` unset or `=duck` → `DuckDbStore` (existing default). +- `CODEHUB_STORE=lbug` → `GraphDbStore` (the new backend). + +`lbug` is the short token the CLI accepts; source-level naming never uses +it as a symbol. The class name is `GraphDbStore`, file names are +`graphdb-adapter.ts` / `graphdb-schema.ts` / `graphdb-pool.ts`, and the +per-kind rel table for the OCH-native `PROCESS_STEP` edge is named +`ProcessStep` in the Cypher schema. The `@ladybugdb/core` npm dependency +is the one allowed surface for the product name in tracked source, and +it is already permitted in `scripts/check-banned-strings.sh` via the +per-literal allowlist (`@ladybugdb[/A-Za-z0-9_-]*`). + +The phased plan, sequenced by milestone dependency per the v1.0 roadmap: + +- **M3** (this milestone): opt-in `CODEHUB_STORE=lbug` ships. DuckDB + stays the default. A byte-identity graphHash parity gate runs on every + CI build so the two backends cannot drift silently. +- **M4 – M6**: both stores stay green. Every new feature that touches + storage writes against `IGraphStore`, not `DuckDbStore` or + `GraphDbStore` directly. +- **M7**: flip the default to `CODEHUB_STORE=lbug` (task T-M7-1). Retain + DuckDB as the legacy backend for temporal-analytics workloads only — + the columnar engine is genuinely better for time-series queries, and + there is no gain in ripping it out. Drop dual-emit `sql|cypher` down + to `cypher`-only at the same time (T-M7-3). + +## Schema choice — polymorphic rel-table-per-edge + +The idiomatic Cypher-on-a-column-store shape is **one named rel table +per edge kind**, each with multiple `FROM / TO` node-type pairs. The ADR +records this explicitly because the v1.0 roadmap originally suggested +one `CodeRelation` rel table with a `type` column, and the LadybugDB +schema docs at +`docs.ladybugdb.com/cypher/data-definition/create-table` make the +opposite recommendation: `CREATE REL TABLE Calls(FROM Function TO +Function, FROM Method TO Method, confidence FLOAT)` gives the planner +the physical partition-by-kind it needs for predicate pushdown, and a +`MATCH (a)-[r:Calls]->(b)` query probes exactly one table with no row +filter. + +Rejected alternative: a single rel table with a `type` property. Reasons +for rejection: + +1. Identical full-scan cost to the DuckDB shape we are leaving — the + planner has nothing to prune. +2. Forces every subsequent query rewrite to include `WHERE r.type = ?`, + which is exactly the idiom the graph engine is supposed to replace. +3. Loses the ability to declare kind-specific constraints (e.g. the + `HAS_METHOD` edge is always `Class → Method` or `Interface → Method`; + the polymorphic shape encodes that constraint in DDL). + +The schema translator in `packages/storage/src/graphdb-schema.ts` emits +one `CREATE REL TABLE` per entry in `getAllRelationTypes()` plus one +`CREATE NODE TABLE CodeNode` with the shared property columns from the +DuckDB `nodes` schema. The parity gate (§graphHash invariant below) +catches any drift between the two schemas at CI time. + +## Concurrency model — process-wide Database + Connection pool + +LadybugDB v0.16's public API exposes one `Database` handle and a pool of +`Connection` objects obtained via `new Connection(db)`. Connections are +**not** safe to call `.query()` on concurrently — two overlapping calls +on the same `Connection` segfault the native binding. The safe shape is +one `Database` opened `READ_WRITE` for the process, plus a pool of +`Connection` objects where each checkout guarantees exclusive use until +it is returned. + +The pool adapter at `packages/storage/src/graphdb-pool.ts` (545 LOC) +encodes that contract. It was lifted from the same-shape adapter in +GitNexus and re-audited for the v0.16 API surface — the LadybugDB +fork's `Connection` lifecycle is materially identical to the Kuzu line +the GitNexus adapter was written against, and the audit did not turn up +a behavioural change that required a rewrite. Parameters (locked in by +AC-M3-2): + +| Parameter | Value | Rationale | +|---|---|---| +| `MAX_CONNS_PER_REPO` | 8 | Matches the concurrent-query budget the MCP tools plan for; beyond 8 the lock contention on the LadybugDB journal becomes the bottleneck anyway. | +| Waiter timeout | 15 s | A queued checkout that waits longer than this surfaces a pool-exhaustion error rather than silently blocking the MCP tool call. | +| Query timeout | 30 s | Mirrors the existing `IGraphStore.query(timeoutMs)` contract. The `sql` MCP tool still enforces its own 5-second default; this ceiling is for long-running CLI paths. | +| Idle sweep | 60 s | Reclaims `Connection` objects that have been checked in but unused, so a quiet repo does not hold 8 native handles open indefinitely. | +| Pool cap | 5 | Upper bound on the **number of distinct `Database` handles** we hold across repos in one process. The v1.0 surface never indexes more than a few repos in one run, so 5 is ample. | + +`W-M3-1` (spec 004) enforces the one-query-per-`Connection` invariant in +tests. The pool queue semantics are covered by the 100-concurrent-read +test under `graphdb-pool.test.ts`. + +## Source naming — no product-name tokens in tracked source + +The banned-strings guardrail (`scripts/check-banned-strings.sh`) +rejects the bare tokens `ladybug` and `kuzu` in tracked source. The +naming strategy below keeps the guardrail clean while still producing +readable, idiomatic TypeScript: + +- Class name: `GraphDbStore` (not a product-name prefix). +- File names: `graphdb-adapter.ts`, `graphdb-schema.ts`, + `graphdb-pool.ts`, `graphdb-adapter.test.ts`, etc. +- OCH-native edge kind `PROCESS_STEP` maps to a Cypher rel table named + `ProcessStep`, **not** the banned GitNexus-style `STEP_IN_PROCESS`. +- The npm dependency `@ladybugdb/core` (declared in + `packages/storage/package.json`) is allowed under a per-literal + allowlist in `scripts/check-banned-strings.sh` — the package scope is + an external identifier, not a source-level symbol. +- This ADR lives under `docs/adr/` and names the product in prose. The + commit that introduces this file also adds `:(exclude)docs/adr` to + the banned-strings `EXCLUDES` list so the historical-rationale prose + does not have to play games with the token boundaries. Source files + are still swept. + +## graphHash invariant and the parity gate + +`graphHash` is computed over the in-memory `KnowledgeGraph` +(`packages/core-types/src/graph-hash.ts`, 45 LOC). It is defined as the +SHA-256 of the canonical-JSON projection `{edges, nodes}` with every +object's keys sorted — the hash function never touches store rows, so +the invariant is **store-agnostic by construction**. + +The parity gate lands at `packages/storage/src/graph-hash-parity.test.ts` +(AC-M3-4, 517 LOC). For every fixture, the test does a symmetric +round-trip through both backends and asserts: + +``` +graphHash(fixture) + === graphHash(rebuildFromDuckDb(duckStore)) + === graphHash(rebuildFromGraphDb(graphDbStore)) +``` + +Three fixtures cover the shape-space: + +- **small**: ≤10 nodes, `DEFINES` + `CALLS` only (sanity shape). +- **medium**: ~60 nodes across `File` / `Class` / `Interface` / `Method` + / `Contributor` with `DEFINES`, `IMPLEMENTS`, `HAS_METHOD`, `CALLS`, + `OWNED_BY`. +- **large**: ≥500 nodes as a long `CALLS` chain with shortcuts, plus a + sweep that emits one edge for every entry in `getAllRelationTypes()` + — so a schema regression that silently drops a rel table surfaces as + a hash mismatch on the next CI run. + +Current runtime is ≈2.1 s across the three fixtures. The budget in spec +004 §AC-M3-4 is 30 s, and the gate is wired into `mise run check` so it +runs on every commit. + +One subtlety the gate codifies: the DuckDB `step` column is `INTEGER +NOT NULL DEFAULT 0`, while the graph-db `step` column is a nullable +`INT32`. When an edge's step is explicitly `0`, the two backends +disagree on readback (DuckDB returns `0`, the graph-db returns `null`). +Both readers in the parity test normalize by dropping `step` when it +reads back as zero or null, which is the same convention +`duckdb-adapter.test.ts` already uses. The fixtures themselves avoid +`step: 0` so the original-graph comparison stays clean. + +## Fallback — Apache AGE on Postgres 18 + +If LadybugDB breaks beyond repair at some point during M3 – M6 (the +library is pre-1.0; a sufficiently-bad ABI break could ship with no +easy fix), the documented escape hatch is Apache AGE on Postgres 18. +AGE is a Cypher extension for Postgres with a comparable data model — a +port would touch the same `IGraphStore` seam this ADR relies on. The +fallback is **documented, not implemented**; the work is scoped as +T-M7-5 and only fires if the primary backend fails the parity gate on +a version we cannot roll back from. + +We pick AGE rather than Neo4j or the cloud-hosted graph engines because +of ADR 0001's rail: self-hosted OSS only, no hosted / managed / SaaS +tier. AGE ships as a Postgres extension and inherits Postgres's +embedded-use patterns directly. Neo4j Community Edition has license +terms (GPLv3) that conflict with our distribution rights under +Apache-2.0 (per the ADR 0001 license filter). + +## 3-phase plan + +| Phase | Milestones | What ships | Default backend | +|---|---|---|---| +| 1 | M3 | Opt-in `CODEHUB_STORE=lbug`, schema translator, pool adapter, parity gate, `sql` MCP tool gains a `cypher` input. | DuckDB. | +| 2 | M4 – M6 | Both backends stay green. Every storage-touching feature writes against `IGraphStore`. Cross-repo federation (M6) exercises both backends end-to-end. | DuckDB. | +| 3 | M7 | Flip default to `CODEHUB_STORE=lbug` (T-M7-1). Keep DuckDB for temporal analytics only (T-M7-2). Drop dual-emit `sql|cypher` down to `cypher`-only (T-M7-3). Final parity audit across the testbed corpus (T-M7-4). | GraphDB. | + +The phased plan is the reason this ADR does not itself flip the default +— that decision belongs to M7, after the second backend has absorbed +two milestones of parallel-work traffic. + +## Risks + +1. **Pre-1.0 library with ABI churn.** `@ladybugdb/core` is at 0.16.1 + as of 2026-05-04. GitNexus pins 0.15.2, so we already know ABI + breaks land every few months. Mitigation: pin the exact minor in + `packages/storage/package.json` (`^0.16.1` today; bumped + intentionally, not via `pnpm up`). The opt-in `CODEHUB_STORE=lbug` + surface means any ABI mismatch shows up cleanly at `GraphDbStore.open()` + time rather than silently corrupting a user's default workflow + (spec 004 state req S-M3-3). +2. **Re-analyze-on-mismatch runbook.** When a user upgrades OCH across + a LadybugDB minor bump, the on-disk database file from the prior + version may refuse to open. `GraphDbStore.open()` surfaces a + specific "database was written by a different `@ladybugdb/core` + version; re-run `codehub analyze --force`" message and does **not** + silently truncate. The runbook is linked from the error text, not + the commit message of the version bump. +3. **Platform support for the native binding.** The library ships + prebuilt binaries for linux-x64, linux-arm64, darwin-x64, and + darwin-arm64 at v0.16.1. CI platforms without a prebuilt binary + will fail the lazy import at `open()`; the parity test skips + cleanly in that case rather than failing the whole run (the + skip-on-missing-binding path is tested explicitly). +4. **Bundled dep vs optional peer.** This ADR hard-depends on + `@ladybugdb/core` (spec 004 §AC-M3-1, user-approved 2026-05-05). + Making it an optional peer was considered and rejected: the parity + test needs the binding in CI, and platform-specific installers + already gate per-OS binaries at the npm level. A missing binary is + a platform issue, not a dependency issue. + +## Status + +- **Proposed**: 2026-05-05 (M3 ADR commit). +- **Accepted**: on merge of `feat/v1-m3-m4` → `main` (the PR that ships + all of M3 at once, per spec 004 AC-M4-8). +- **Superseded**: not before M7. M7 adds a follow-up ADR (scope: flip + default + drop SQL dual-emit + final parity audit). + +## References + +- `docs/adr/0001-storage-backend.md` — the DuckDB selection that this + ADR leaves in place as the M3 – M6 default. +- `.erpaval/ROADMAP.md` §M3, §M7 — the durable roadmap rows that + sequence this work. +- `.erpaval/specs/004-m3-m4/spec.md` §AC-M3-1..6 — acceptance criteria + landed in Wave 1 + Wave 2. +- `packages/core-types/src/graph-hash.ts` — store-agnostic hash + definition. +- `packages/storage/src/graph-hash-parity.test.ts` — parity gate, three + fixtures (small / medium / large), 24-edge-kind sweep. +- `packages/storage/src/graphdb-pool.ts` — pool adapter, 545 LOC, + lifted and re-audited from the GitNexus adapter. +- `packages/storage/src/graphdb-schema.ts` — polymorphic + rel-table-per-kind DDL translator. +- `scripts/check-banned-strings.sh` — guardrail; this ADR's commit + adds `:(exclude)docs/adr` to the `EXCLUDES` list so + architectural-history prose can name the tool. +- LadybugDB schema docs (cited above) — + `docs.ladybugdb.com/cypher/data-definition/create-table`. + +## Provenance + +LadybugDB is the community successor to the Kuzu project. Kuzu was +acquired by Apple in early 2026 and its public-source cadence stopped; +LadybugDB forked from the pre-acquisition open-source codebase under +the existing permissive license and continues development under the +`@ladybugdb/core` npm identifier. Pinning a specific minor is a hard +requirement in both pre-acquisition and post-fork lineages — this ADR +does not rely on any capability that was added to Kuzu after the fork +point, and the fork's schema surface (named rel tables, Cypher +dialect, native `Database` + `Connection` API) is 1:1 compatible with +the pre-acquisition docs. We cite the LadybugDB docs URL in the schema +section above because that is the current authoritative reference; the +Kuzu docs for the same surface are equivalent for our purposes but are +not guaranteed to stay online. diff --git a/packages/cli/src/cobol-proleap-setup.test.ts b/packages/cli/src/cobol-proleap-setup.test.ts new file mode 100644 index 00000000..fb39c9c1 --- /dev/null +++ b/packages/cli/src/cobol-proleap-setup.test.ts @@ -0,0 +1,193 @@ +/** + * Tests for `codehub setup --cobol-proleap`. Uses an in-memory ProcessApi + * so the suite never shells out. Covers: + * + * - Missing tool precondition errors emit tool-specific install hints. + * - javac < 17 refused with the JDK-upgrade hint. + * - Happy path: git clone + mvn install + javac + atomic rename succeed; + * the result reports the final JAR + wrapper class paths. + * - Idempotency: a second call with the JAR + wrapper class already in + * place skips without re-running the build. + */ + +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { + DEFAULT_PROCESS_API, + defaultVendorDir, + type ProcessApi, + type ProcessResult, + runSetupCobolProleap, +} from "./cobol-proleap-setup.js"; + +/** Scripted ProcessApi: looks up `(cmd, args)` in the registered map. */ +interface Script { + toolResponses: Map; + fsFiles: Set; + fsDirs: Set; + fsReaddir: Map; + calls: { cmd: string; args: readonly string[] }[]; +} + +function makeScript(init: Partial