From c0239b4160982e6b5b589bdc3cc5467ae38ca516 Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 20:14:21 +0000 Subject: [PATCH 01/14] chore(brainstorm): source-pipeline-integrity --- .../cli-output/claude.md | 38 +++++ .../cli-output/prompt.md | 88 +++++++++++ .../source-pipeline-integrity/design.md | 135 ++++++++++++++++ .../source-pipeline-integrity/plan.md | 146 ++++++++++++++++++ .../source-pipeline-integrity/variants.md | 93 +++++++++++ 5 files changed, 500 insertions(+) create mode 100644 docs/brainstorm/source-pipeline-integrity/cli-output/claude.md create mode 100644 docs/brainstorm/source-pipeline-integrity/cli-output/prompt.md create mode 100644 docs/brainstorm/source-pipeline-integrity/design.md create mode 100644 docs/brainstorm/source-pipeline-integrity/plan.md create mode 100644 docs/brainstorm/source-pipeline-integrity/variants.md diff --git a/docs/brainstorm/source-pipeline-integrity/cli-output/claude.md b/docs/brainstorm/source-pipeline-integrity/cli-output/claude.md new file mode 100644 index 00000000..9770b165 --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/cli-output/claude.md @@ -0,0 +1,38 @@ +### Variant 1: Two shared kernels, cluster-local units + +- **Approach**: Extract exactly two new shared abstractions, each riding in with the first unit that needs it: a gitignore-style path-scope engine (`src/core/fs/ignore.ts`: nested ignore-file composition, nearer-`!`-wins, `.git/info/exclude` layering) introduced by Unit 1 and consumed by Unit 2's `--src-subpath`/`--exclude`; and a diagnostics-signal model (issue class + detector + optional fixer + next-command hint) introduced by Unit 9 and consumed by Unit 10's snapshot. Everything else lands as local changes to its existing module, organized into four clusters: scope/gate (1→2→3), ingest integrity (5, 4), provenance/query (6, 7, 8), operator surface (11, 9→10). The dependency spine is short - only 1→2 and 9→10 are hard edges; ship order 11, 1, 2, 3, 5, 4, 6, 7, 8, 9, 10 keeps every prefix coherent. +- **Trade-offs**: + - Pro: shared logic gets exactly one home (ignore matching, issue/hint model) without any infrastructure-only commit - each abstraction ships inside a feature commit with its tests. + - Pro: maximal prefix-shippability; a stalled wave still leaves `main` releasable after any unit. + - Pro: byte-identical opt-out is easy - the ignore engine replaces the static skip-dir list only where Unit 1 wires it; nothing else changes uninvoked. + - Con: the ingest path (Units 2, 3, 5, 4) stays four separate touch points in `ingest.ts`/`batch-plan.ts` rather than one explicit pipeline; a future wave may still want that consolidation. + - Con: the diagnostics model's shape is decided by Unit 9's needs and may need a small extension when Unit 10 arrives. +- **Complexity**: medium +- **Risk**: low + +### Variant 2: Source-pipeline spine + +- **Approach**: Treat Units 1-5 as stages of one explicit discovery pipeline abstraction in `src/core/brain/ingest/`: scope (ignore engine + subpath/exclude) → gate (`extractable` check) → pre-extract (deterministic code-structure pass) → dispatch → reconcile, each stage a typed hook on a shared `SourcePipeline` context. Operator tooling (9, 10) similarly reads a shared health-signal registry, and hygiene scan (Unit 1) becomes another consumer of the same scope stage. Units 6, 7, 8, 11 remain peripheral standalone changes. +- **Trade-offs**: + - Pro: strongest conceptual match to the wave theme - one choke point for "what enters the vault and how," which future ingest features slot into. + - Pro: reconciliation (Unit 4) and gating (Unit 3) become trivially testable as pipeline stages rather than call-site patches. + - Con: the pipeline scaffold is effectively an infrastructure commit, which the repo's conventions discourage, and it forces refactoring `ingest.ts`/`batch-plan.ts` before any user-visible unit ships - a stalled wave leaves a half-migrated pipeline. + - Con: highest risk to the byte-identical opt-out constraint, since existing ingest flow is rerouted through new machinery even when no new flag is set. + - Con: hygiene scan (Unit 1) importing from `brain/ingest` inverts sensible layering; the ignore engine would need to live below both anyway, eroding the "one spine" elegance. +- **Complexity**: large +- **Risk**: high + +### Variant 3: Eleven islands, convention-only sharing + +- **Approach**: Every unit lands independently in its own existing module with no new shared homes: Unit 1 implements ignore parsing inside `hygiene/`, Unit 2 imports that parser directly (a hygiene→ingest reach-across), Unit 10 calls the existing verb functions (`brain_health`, `brain_doctor`, etc.) and formats their outputs itself, and Unit 9 pairs fixers with doctor checks ad hoc inside `doctor.ts`. Ordering is nearly free - only Unit 2's import of Unit 1's parser is a hard edge. +- **Trade-offs**: + - Pro: cheapest to execute and the easiest to parallelize across agents; every commit is small, atomic, and independently revertable. + - Pro: near-perfect prefix-shippability with zero scaffolding to leave half-finished. + - Con: the ignore engine living under `hygiene/` while `ingest/` imports it violates the one-directional-layering direction, or else gets duplicated - either way conflicting with "shared logic gets exactly one home." + - Con: Unit 10 re-deriving and re-formatting signals from six verbs duplicates presentation logic and drifts as those verbs evolve; hints get hardcoded per call site instead of traveling with the issue. + - Con: defers exactly the consolidation debt the post-v1.30.1 refactor just paid down. +- **Complexity**: small +- **Risk**: medium + +### Recommended: Variant 1 +**Rationale**: Variant 1 delivers the two abstractions this wave genuinely reuses (ignore composition, diagnostics signals with hints) in their correct one-home locations, while avoiding Variant 2's infrastructure-first commit and its threat to the byte-identical opt-out and atomic-feature-commit conventions. Unlike Variant 3, it keeps layering one-directional - the ignore engine sits below both hygiene and ingest, and hints travel with issue definitions instead of being duplicated in the snapshot. Its short dependency spine (1→2, 9→10) preserves the constraint that a partially completed wave still ships as a coherent prefix of v1.34.0. diff --git a/docs/brainstorm/source-pipeline-integrity/cli-output/prompt.md b/docs/brainstorm/source-pipeline-integrity/cli-output/prompt.md new file mode 100644 index 00000000..ef238027 --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/cli-output/prompt.md @@ -0,0 +1,88 @@ +You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation. + +# Task + +This is a release wave of eleven kanban tasks for Open Second Brain (OSB). The wave theme is source pipeline integrity and operator tooling: everything about getting sources into the vault correctly (scoped, gated, reconciled, deterministically pre-extracted, provenance-carrying), querying what is there, and keeping the vault healthy. Brainstorm the wave architecture as a whole: where shared abstractions live, what the dependency spine is, and how the units cluster. Do not judge whether individual units are worth doing; all eleven ship. + +## Unit 1 (t_9654de80) - Nested .gitignore in hygiene file scan +Teach the hygiene file scan to honor `.gitignore` (and a project-local ignore file) with git-like nested composition: an ignore file deeper in the tree applies to its own subtree, and a nearer `!` re-include wins, replacing the single hardcoded skip-dir list. Code today: `src/core/hygiene/scan-repo.ts:75` uses a static skip-dir array; no gitignore parsing exists anywhere in the scan. The nearer-`!`-wins composition rule is the deliberate part; compose with `.git/info/exclude` semantics. + +## Unit 2 (t_e82101a5) - `--src-subpath` and `--exclude` on source ingest +Add `--src-subpath` and `--exclude` to source ingest/sync so a monorepo can be ingested from one subdirectory, excluding sibling packages. `--exclude` should compose with ignore handling rather than duplicate it. Incremental sync exists via `o2b watch` (command-manifest.ts:259); no subdir scoping anywhere today. + +## Unit 3 (t_ed856388) - Honor the schema `extractable` flag during page discovery +OSB already stores an `extractable` flag on packs via schema mutation (schema-mutate.ts:54 `set_extractable`, handler :260-263) but nothing consults it during page discovery/extraction. Add discovery-time enforcement: pages in non-extractable packs are skipped up front. Behavior-only wiring of an existing flag. + +## Unit 4 (t_d067a153) - Reconcile dispatched-vs-ingested documents in batch ingest +Large-folder ingest plans bounded batches (`planBatches`, src/core/brain/ingest/batch-plan.ts:125) and folds per-source completion into a checkpoint (`recordCompleted`, checkpoint.ts:169) and content manifest. Nothing compares the plan's dispatched set against the files that actually came back. Add a `reconcilePlan(planId)` helper next to the checkpoint, invoked after a batch/plan drains, warning on silently-lost sources. + +## Unit 5 (t_ef786747) - Local no-LLM code-structure extractor as a pre-ingest pass +A deterministic, no-LLM code-structure extractor that parses code sources into classes/functions/imports/inheritance edges as JSON, run as a pre-pass before agent-driven ingest to cut extraction tokens and produce deterministic entity/edge seeds. Must stay pure-stdlib for the default path. Scope it as a lightweight fallback pre-pass, complementing (not replacing) the detect-only codegraph partner integration (src/core/partner/codegraph.ts). Ingest consumer: src/core/brain/ingest/ingest.ts. + +## Unit 6 (t_a3d1adb0) - Inline `[Source: , YYYY-MM-DD]` prose citations into the temporal timeline +Parse citation markers embedded in note prose and promote each into the temporal timeline as a dated provenance entry. Timeline builds only from structured logs today (temporal/build-index.ts walks Brain/log/*.jsonl); provenance lives in provenance/provenance.ts + portability/origins.ts. New parser feeding existing temporal + provenance sinks; handle malformed dates and dedup against already-logged source events. NOTE the OSB constraint: no hardcoded natural-language word lists; the `[Source: ...]` marker is a structural syntax, which is acceptable, but date parsing must be structural. + +## Unit 7 (t_618f7211) - Configurable FTS tokenizer language/diacritic rules +The FTS tokenizer is hardcoded (`tokenize='unicode61 remove_diacritics 2'` in search/schema.ts). Expose tokenizer/language (stemming + diacritic rules) as configuration so non-English vaults index on language-appropriate rules, leaving the default intact. CJK is handled out-of-band via a separate fts_content column + trigram prefilter; the config must compose with that path, not replace it. Reindex command already exists (`o2b search reindex`). Config must not require language word lists. + +## Unit 8 (t_9bee8f0b) - Graph-degree cardinality predicates in the search/filter DSL +Add a cardinality predicate to the user-facing search/filter DSL filtering notes by the COUNT of link-graph relations: backlinks/outlinks `= 0` (orphans/leaves), `>= N` (hubs). Degree is already computed internally (link-graph/graph-index.ts:33 `degree` map, moc-audit backlink buckets); the DSL (src/core/search/property-filter.ts) matches only frontmatter key/values today. CLI surface in src/cli/search.ts. + +## Unit 9 (t_bd6cc4cb) - `o2b brain doctor --repair` +Add a guarded repair mode to the existing read-only doctor (doctor.ts) that performs targeted fixes for the issue classes doctor already detects (orphaned references, WAL gaps). Opt-in, dry-run/preview by default, `--strict` read-only behavior preserved as default. + +## Unit 10 (t_9f9c5466) - Unified operator status snapshot +One readable CLI health snapshot combining counts, stale + orphaned pages, review queue depth, active profile, and state-file health, with an actionable next-command hint printed on each problem line. The raw signals are already computed across brain_health, brain_doctor, brain_hygiene, brain_stale_scan, brain_review_candidates, brain_brief operator view; ship the consolidation + per-line hints only. + +## Unit 11 (t_2ed754d1) - Early-closed stdout pipe as clean exit in CLIs +Make `o2b` and `vault-log` treat an early-closed stdout pipe (EPIPE after `| head`) as a normal exit 0 instead of a nonzero failure. Entry points are bash wrappers exec-ing `bun run src/cli/main.ts`; main exits via `main(...).then((code) => process.exit(code))` (src/cli/main.ts:933). No EPIPE/SIGPIPE handling exists. Small and self-contained. + +# Project context + +Open Second Brain: TypeScript on Bun, CLI (`o2b`) + MCP server over an Obsidian-compatible Markdown vault, bun:sqlite + sqlite-vec hybrid search. Deterministic kernel: the core never calls an LLM; agent-driven steps live outside the kernel. + +Recent commits: +77513f2b feat: belief lifecycle and decision memory (v1.33.0) (#141) +61e93d24 fix(config): derive vault store reference from a keyed installation secret (#140) +9a649dd6 feat: memory write-path integrity and store safety wave (v1.32.0) (#139) +f2a037eb feat: today operator surface - dashboard, open loops, marker write-back (v1.31.0) (#138) +13bde6c3 refactor: remove all import cycles, decompose search.ts (v1.30.1) (#137) +fd5661f9 feat: governance visibility - vitals scorecard + batch-inflation lint (v1.30.0) (#136) +a99b0e71 feat(brain): add o2b brain vitals scorecard + batch-concept-inflation lint (#135) +70fb36e1 feat: operability, safety & first-run experience (v1.29.0) (#134) +ac26a675 feat: retrieval & ranking quality (v1.28.0) (#133) +5cd52e70 fix(hermes): resolve o2b when memory provider PATH is tiny (v1.27.1) (#131) + +Related files: +src/core/hygiene/scan-repo.ts, src/core/brain/ingest/{ingest.ts,batch-plan.ts,checkpoint.ts,content-manifest.ts}, src/core/schema/schema-mutate.ts, src/core/temporal/build-index.ts, src/core/provenance/provenance.ts, src/core/search/{schema.ts,property-filter.ts,fts.ts}, src/core/brain/link-graph/graph-index.ts, src/cli/main.ts, src/cli/search.ts, src/cli/brain/verbs/*, src/mcp/*, src/core/partner/codegraph.ts, scripts/o2b, scripts/vault-log + +Conventions: +- Each unit lands as one atomic conventional feature commit with its tests; infrastructure-only commits are discouraged. +- Post-v1.30.1 direction: shared choke points, one-directional layering, no import cycles; shared logic gets exactly one home. +- SOLID, KISS, DRY; constants extracted; typed errors surfaced explicitly; no silent do-nothing fallbacks; no stubs. +- Language-agnostic: no built-in natural-language word lists anywhere; structural signals and config-supplied vocabularies only. +- New CLI verbs get an MCP counterpart when they are agent-relevant; MCP tool descriptions are capped and parity-tested. +- Byte-identical opt-out: with new features unconfigured, existing outputs must not change. + +Constraints: +- Bun runtime, bun:sqlite; no new external dependencies for the default path (pure stdlib for parsing/extraction). +- Do not change existing public APIs; new params optional. +- The deterministic kernel calls no LLM. +- The wave ships as one PR / one release (v1.34.0); a partially completed wave must still ship coherently as a prefix. + +# Required output format + +Produce exactly 3 distinct architectural variants. For each variant: + +### Variant N: +- **Approach**: 2-3 sentences describing the variant. +- **Trade-offs**: bullet list of pros and cons. +- **Complexity**: small | medium | large +- **Risk**: low | medium | high + +After the three variants, add exactly one recommendation: + +### Recommended: Variant N +**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above. + +Output nothing outside of these sections. diff --git a/docs/brainstorm/source-pipeline-integrity/design.md b/docs/brainstorm/source-pipeline-integrity/design.md new file mode 100644 index 00000000..cdce167b --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/design.md @@ -0,0 +1,135 @@ +# Source pipeline integrity and operator tooling - one wave, eleven units + +**Status:** accepted +**Author:** Claude orchestrator (via feature-release-playbook) +**Audience:** implementation + +## Problem statement + +Sources enter the vault through paths that cannot be scoped or gated, batches +can silently lose documents, prose citations never reach the temporal layer, +and vault health signals are scattered across six read-only surfaces with no +repair action. This wave makes intake trustworthy end to end: scoped and +gated discovery, reconciled dispatch, deterministic pre-extraction, citation +provenance, configurable indexing, degree-aware querying, and an operator +toolchain that can both summarize and repair. + +## Scope + +Eleven kanban tasks ship as one PR and one release (v1.34.0): + +- `t_2ed754d1` early-closed stdout pipe exits clean in `o2b` / `vault-log` +- `t_9654de80` nested `.gitignore` composition in the hygiene file scan +- `t_e82101a5` `--src-subpath` / `--exclude` scoping on source ingest +- `t_ed856388` `extractable` flag honored during page discovery +- `t_ef786747` local no-LLM code-structure extractor as a pre-ingest pass +- `t_d067a153` dispatched-vs-ingested reconciliation for batch ingest plans +- `t_a3d1adb0` inline `[Source: , YYYY-MM-DD]` citations become dated + provenance events on the temporal timeline +- `t_618f7211` configurable FTS tokenizer language and diacritic rules +- `t_9bee8f0b` graph-degree cardinality predicates in the search/filter DSL +- `t_bd6cc4cb` `o2b brain doctor --repair` for the issue classes doctor + already detects +- `t_9f9c5466` unified operator status snapshot with per-problem + next-command hints + +## Out of scope + +- Rebuilding the reindex command (exists as `o2b search reindex`). +- Replacing or extending the codegraph partner integration; the pre-ingest + extractor is a fallback pre-pass, not a codegraph substitute. +- Tree-sitter or any non-stdlib parsing backend for the extractor. +- An interactive profile explorer or onboarding scaffolder (different theme). +- Repair actions for issue classes doctor does not already detect. +- `t_71f96533` (created_at bounds on raw-session grep) is excluded as a + duplicate: v1.33.0 already shipped `since` / `before` on session grep. + +## Chosen approach + +Consultant Variant 1: two shared kernels, cluster-local units. Exactly two +new shared abstractions, each riding inside the first feature commit that +needs it: + +1. **Path-scope engine** `src/core/fs/ignore.ts` - nested ignore-file + composition with git semantics (deeper file scopes its subtree, + nearer `!` re-include wins, `.git/info/exclude` layering). Introduced by + the hygiene-scan unit (P1), consumed by ingest scoping (P2). It sits + below both `hygiene/` and `brain/ingest/`, keeping layering + one-directional. +2. **Diagnostics-signal model** - issue class + detector + optional fixer + + next-command hint, introduced by `doctor --repair` (O2) and consumed by + the unified status snapshot (O3), so hints travel with issue definitions + instead of being duplicated in the snapshot formatter. + +Every other unit is a local change to its existing module. Four clusters: +scope/gate (P1 -> P2 -> P3), ingest integrity (P4, P5), provenance/query +(Q1, Q2, Q3), operator surface (O1, O2 -> O3). Hard dependency edges are +only P1 before P2 and O2 before O3; every prefix of the ship order leaves +`main` releasable. + +## Design decisions + +- **Ignore engine location** `src/core/fs/ignore.ts`, not under `hygiene/` + or `ingest/`: both consume it; one home, no reach-across imports. +- **Byte-identical opt-out everywhere**: unset config, absent flags, and + unconfigured features must leave existing outputs unchanged; + regression-tested per unit. +- **Explicit errors, no silent fallbacks**: malformed ignore patterns, + invalid tokenizer configs, invalid degree predicates, malformed repair + targets, and unknown citation dates surface as typed errors or explicit + warnings; nothing is silently skipped without a report. +- **Reconciliation is a report, not a retry**: `reconcilePlan(planId)` diffs + the dispatched set against completed entries and returns/warns the gap; + it does not re-dispatch (that stays operator-driven). +- **Extractor stays deterministic and stdlib-only**: structural parsing per + language family via regex/line grammar, emitting JSON entity/edge seeds + that ingest passes to the agent as pre-extracted facts; no natural-language + word lists, no LLM in the kernel. +- **Citation syntax is structural**: `[Source: , YYYY-MM-DD]` is a + fixed marker grammar with an ISO-shaped date; no language-specific parsing. + Dedup key is (normalized name, date) against already-logged source events. +- **Tokenizer config composes with CJK**: config selects FTS5 tokenizer + options (stemming/diacritics); the existing trigram prefilter path is + untouched. Changing the config requires an explicit `o2b search reindex`; + the CLI says so rather than reindexing implicitly. +- **Degree predicates read the existing graph index**: the DSL gains count + predicates over backlinks/outlinks backed by `graph-index.ts` degree data; + no new graph computation. +- **Repair is opt-in and previewed**: `doctor --repair` defaults to dry-run + preview; `--apply` performs fixes; `--strict` read-only behavior is + unchanged. Every applied fix logs a typed event. +- **EPIPE handling is scoped to stdout write failures**: an early-closed + stdout pipe exits 0; all other I/O errors keep failing loudly. +- **MCP parity**: agent-relevant new surfaces (status snapshot, repair + preview, degree predicates, citation scan) get MCP counterparts; frozen + parity lists and tool-count tests update accordingly. + +## File changes + +New: `src/core/fs/ignore.ts`, `src/core/brain/ingest/pre-extract.ts` (code +structure extractor), `src/core/brain/ingest/reconcile.ts`, +`src/core/temporal/citations.ts`, diagnostics-signal module next to +`doctor.ts`, snapshot verb module, tests per unit. + +Modified: `src/core/hygiene/scan-repo.ts`, ingest CLI/MCP surface and +`ingest.ts` / `batch-plan.ts` / `checkpoint.ts`, discovery path consuming +`extractable`, `src/core/search/schema.ts` + config for tokenizer, +`src/core/search/property-filter.ts` + `src/cli/search.ts` for degree +predicates, `src/cli/main.ts` / `scripts/*` for EPIPE, doctor CLI/MCP, +docs and manifests (version 1.34.0). + +## Risks and open questions + +- Gitignore composition has sharp edge cases (anchoring, directory-only + patterns, `!` precedence); mitigate with a property/table test suite + mirroring git's documented rules. Full `git check-ignore` parity is not + claimed; the supported subset is documented. +- The extractor's language coverage is bounded; unknown languages must fall + through to today's behavior explicitly (reported as unextracted, never a + fake empty result). +- Repair fixers touch vault files; every fixer needs a preview diff and an + idempotency test. +- The diagnostics-signal model shape is set by O2 and may need a small + extension when O3 lands; acceptable churn inside one branch. +- MCP tool-count and description-budget guards will need deliberate updates; + keep additions within the 300-char description cap. diff --git a/docs/brainstorm/source-pipeline-integrity/plan.md b/docs/brainstorm/source-pipeline-integrity/plan.md new file mode 100644 index 00000000..e61e2fa3 --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/plan.md @@ -0,0 +1,146 @@ +# Source pipeline integrity and operator tooling - implementation plan + +Ordering follows consultant Variant 1: two shared kernels ride inside the +first feature commit that needs them (P1 owns `src/core/fs/ignore.ts`, O2 +owns the diagnostics-signal model); every other unit is a local change to +its existing module. Hard dependency edges are only P1 before P2 and O2 +before O3; every prefix of the sequence leaves the branch releasable. +Each unit lands as one atomic conventional commit with its tests, formatted +(oxfmt) and lint-clean (oxlint, 134-warning/0-error baseline). + +Sequence: O1 -> P1 -> P2 -> P3 -> P4 -> P5 -> Q1 -> Q2 -> Q3 -> O2 -> O3 -> L + +## Tasks + +### Task O1 - clean exit on early-closed stdout pipe (`t_2ed754d1`, p3, standalone) +- **Files**: `src/cli/main.ts` (EPIPE-aware exit path), shared stdout-write + guard if needed by `vault-log`, tests. +- **Acceptance**: `o2b | head -1` exits 0 with no error + output for both `o2b` and `vault-log` entry points; EPIPE on stdout maps + to exit 0; any other I/O error still fails nonzero with its message; a + regression test simulates a closed-pipe write. +- **Depends on**: none. + +### Task P1 - nested gitignore composition in hygiene scan (`t_9654de80`, p3, path-scope anchor) +- **Files**: new `src/core/fs/ignore.ts` (ignore-rule engine: pattern parse, + nested composition, nearer-`!`-wins, `.git/info/exclude` layering), + `src/core/hygiene/scan-repo.ts` wiring, tests (table suite mirroring git + documented semantics). +- **Acceptance**: the hygiene scan skips paths ignored by root and nested + `.gitignore` files; a deeper ignore file scopes only its subtree; a nearer + `!` re-include wins over an outer ignore; `.git/info/exclude` participates; + repos without ignore files scan byte-identically to today (static skip-dir + behavior preserved as baseline); malformed patterns surface as explicit + warnings, never silent skips. +- **Depends on**: none. + +### Task P2 - source ingest scoping flags (`t_e82101a5`, p3) +- **Files**: ingest CLI/MCP surface (`--src-subpath`, `--exclude`), + `src/core/brain/ingest/*` walk scoping via `src/core/fs/ignore.ts` + pattern matching, tests. +- **Acceptance**: ingest of a monorepo scoped with `--src-subpath pkg/a` + processes only that subtree; `--exclude` patterns compose with ignore + handling (same engine, no duplicate matcher); a subpath outside the source + root rejects with a typed error; without the new flags ingest planning is + byte-identical to today. +- **Depends on**: P1 (`src/core/fs/ignore.ts`). + +### Task P3 - extractable flag gate in discovery (`t_ed856388`, p3) +- **Files**: page-discovery path (locate the pack iteration), gate consuming + the existing `extractable` schema flag, tests. +- **Acceptance**: pages in packs marked non-extractable are skipped before + extraction and reported as skipped-with-reason in the discovery result; + packs without the flag behave exactly as today; the skip is logged, not + silent; no schema mutation surface changes. +- **Depends on**: none. + +### Task P4 - no-LLM code-structure pre-ingest extractor (`t_ef786747`, p3) +- **Files**: new `src/core/brain/ingest/pre-extract.ts` (deterministic + stdlib-only structural extractor: classes/functions/imports/inheritance + edges as JSON seeds), wiring in `src/core/brain/ingest/ingest.ts` to pass + seeds to the agent step, CLI/MCP exposure of the pre-pass, tests. +- **Acceptance**: running the pre-pass on a TypeScript/JavaScript and Python + source produces deterministic JSON entity/edge seeds (same input, same + output); unknown languages are reported as unextracted, never a fake empty + success; ingest with the pre-pass off is byte-identical to today; no + natural-language word lists; no new dependencies. +- **Depends on**: none (P2's scoping optional at call site). + +### Task P5 - dispatched-vs-ingested reconciliation (`t_d067a153`, p3) +- **Files**: new `src/core/brain/ingest/reconcile.ts` (`reconcilePlan` + keyed on `planId` diffing `BatchPlan` dispatched set vs checkpoint + completed set), invocation after a batch/plan drains, CLI/MCP surfacing of + the gap report, tests. +- **Acceptance**: a plan whose agent response omits dispatched files yields + a reconciliation report naming each lost source; a fully completed plan + reports an empty gap; the report is a warning surface, not a retry; the + reconcile is idempotent and read-only over checkpoint state. +- **Depends on**: none. + +### Task Q1 - inline citation promotion to temporal timeline (`t_a3d1adb0`, p3) +- **Files**: new `src/core/temporal/citations.ts` (structural + `[Source: , YYYY-MM-DD]` marker parser + promotion into temporal + log/provenance sinks), scan surface (CLI/MCP), tests. +- **Acceptance**: a note containing a well-formed citation marker produces a + dated provenance event on the timeline; re-scanning does not duplicate + (dedup on normalized name + date against already-logged source events); + malformed markers (bad date shape, missing comma) are reported explicitly + and skipped; notes without markers produce no events; parsing is purely + structural. +- **Depends on**: none. + +### Task Q2 - configurable FTS tokenizer (`t_618f7211`, p3) +- **Files**: `src/core/search/schema.ts` (tokenizer string assembled from + config), config keys (e.g. `search.fts_tokenizer.*`), validation, docs + note that changing config requires `o2b search reindex`, tests. +- **Acceptance**: with no config the generated schema keeps + `unicode61 remove_diacritics 2` byte-identically; valid config changes the + tokenizer clause after reindex; invalid tokenizer options reject with a + typed error listing allowed values; the CJK trigram path is untouched + (its tests stay green); no implicit reindex. +- **Depends on**: none. + +### Task Q3 - graph-degree predicates in filter DSL (`t_9bee8f0b`, p3) +- **Files**: `src/core/search/property-filter.ts` (count predicates over + backlinks/outlinks), degree lookup via existing + `src/core/brain/link-graph/graph-index.ts` data, `src/cli/search.ts` and + MCP filter surface, tests. +- **Acceptance**: filters can select notes by backlink/outlink count with + `=`, `!=`, `>`, `>=`, `<`, `<=` (orphans `= 0`, hubs `>= N`); results match + the graph index degree data; invalid predicate syntax rejects with a typed + error; queries without degree predicates behave byte-identically. +- **Depends on**: none. + +### Task O2 - doctor repair mode (`t_bd6cc4cb`, p3, diagnostics anchor) +- **Files**: diagnostics-signal model (issue class + detector + optional + fixer + next-command hint) co-located with `src/core/brain/doctor.ts`, + `--repair` (dry-run preview default) and `--apply` on the doctor CLI/MCP, + fixers only for issue classes doctor already detects (orphaned references, + WAL gaps), typed event per applied fix, tests. +- **Acceptance**: `doctor --repair` previews planned fixes without writing; + `--repair --apply` performs them and logs one typed event per fix; + re-running after apply is a no-op (idempotent); `--strict` and plain + doctor remain read-only and byte-identical; unfixable issue classes state + so explicitly. +- **Depends on**: none. + +### Task O3 - unified operator status snapshot (`t_9f9c5466`, p3) +- **Files**: snapshot verb (CLI + MCP) composing existing signal sources + (health, doctor, hygiene, stale scan, review candidates, active profile, + state-file health) through the diagnostics-signal model, per-problem + next-command hint rendering, tests. +- **Acceptance**: one command prints a consolidated readable snapshot with + counts, stale/orphaned, review queue depth, active profile, and state + health; every problem line carries the exact next command to run (hint + supplied by the signal definition, not hardcoded in the formatter); a + healthy vault prints a compact all-clear; snapshot performs reads only. +- **Depends on**: O2 (diagnostics-signal model). + +### Task L - docs, CHANGELOG, version bump +- **Files**: `README.md`, `CHANGELOG.md` (`## [1.34.0]` + link reference), + `docs/cli-reference.md`, `docs/mcp.md`, `package.json` 1.34.0 + + `bun run scripts/sync-version.ts`. +- **Acceptance**: one CHANGELOG entry covers all eleven units; + `bun run sync-version:check` passes; README and reference docs cover the + new surfaces. +- **Depends on**: all previous tasks. diff --git a/docs/brainstorm/source-pipeline-integrity/variants.md b/docs/brainstorm/source-pipeline-integrity/variants.md new file mode 100644 index 00000000..bb5d9f26 --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/variants.md @@ -0,0 +1,93 @@ +# Source pipeline integrity and operator tooling - variant audit trail + +Consultant: Claude Code (`claude -p`), prompt in `cli-output/prompt.md`, raw +output in `cli-output/claude.md`. Three variants were requested for the wave +architecture as a whole (where shared abstractions live), not for the worth +of individual units. The fallback consultant (Codex) was not invoked because +the primary run succeeded with three parseable variants. + +## Variant 1: Two shared kernels, cluster-local units + +- **Approach**: extract exactly two new shared abstractions, each riding in + with the first unit that needs it - a gitignore-style path-scope engine + (`src/core/fs/ignore.ts`) introduced by the hygiene-scan unit and consumed + by ingest scoping, and a diagnostics-signal model (issue class + detector + + optional fixer + next-command hint) introduced by `doctor --repair` and + consumed by the status snapshot. Everything else lands as local changes to + its existing module, in four clusters with a short dependency spine. +- **Trade-offs**: + - Pro: shared logic gets exactly one home without any infrastructure-only + commit; each abstraction ships inside a feature commit with tests. + - Pro: maximal prefix-shippability; a stalled wave leaves `main` + releasable after any unit. + - Pro: byte-identical opt-out is easy; nothing changes uninvoked. + - Con: the ingest path stays four separate touch points rather than one + explicit pipeline; a future wave may want that consolidation. + - Con: the diagnostics model shape is set by the repair unit and may need + a small extension when the snapshot unit arrives. +- **Complexity**: medium +- **Risk**: low + +## Variant 2: Source-pipeline spine + +- **Approach**: treat the five intake units as stages of one explicit + discovery pipeline abstraction (`SourcePipeline` context with typed hooks: + scope, gate, pre-extract, dispatch, reconcile); operator tooling reads a + shared health-signal registry; remaining units stay peripheral. +- **Trade-offs**: + - Pro: strongest conceptual match to the wave theme; one choke point for + what enters the vault. + - Pro: reconciliation and gating become trivially testable pipeline + stages. + - Con: the pipeline scaffold is an infrastructure commit the repo's + conventions discourage, and it forces refactoring `ingest.ts` / + `batch-plan.ts` before any user-visible unit ships; a stalled wave + leaves a half-migrated pipeline. + - Con: highest risk to byte-identical opt-out, since existing ingest is + rerouted through new machinery even when no new flag is set. + - Con: hygiene importing from `brain/ingest` inverts layering; the ignore + engine must live below both anyway. +- **Complexity**: large +- **Risk**: high + +## Variant 3: Eleven islands, convention-only sharing + +- **Approach**: every unit lands independently in its own existing module + with no new shared homes; ignore parsing lives inside `hygiene/` and + ingest imports it across layers; the snapshot re-formats six verbs' + outputs itself; fixers pair with doctor checks ad hoc. +- **Trade-offs**: + - Pro: cheapest to execute, easiest to parallelize, every commit small + and independently revertable. + - Pro: near-perfect prefix-shippability with zero scaffolding. + - Con: the ignore engine under `hygiene/` with `ingest/` importing it + violates one-directional layering, or gets duplicated. + - Con: the snapshot re-deriving signals from six verbs duplicates + presentation logic and drifts; hints get hardcoded per call site. + - Con: defers exactly the consolidation debt the post-v1.30.1 refactor + paid down. +- **Complexity**: small +- **Risk**: medium + +## Consultant recommendation + +Variant 1. "Variant 1 delivers the two abstractions this wave genuinely +reuses (ignore composition, diagnostics signals with hints) in their correct +one-home locations, while avoiding Variant 2's infrastructure-first commit +and its threat to the byte-identical opt-out and atomic-feature-commit +conventions. Unlike Variant 3, it keeps layering one-directional - the +ignore engine sits below both hygiene and ingest, and hints travel with +issue definitions instead of being duplicated in the snapshot. Its short +dependency spine (1->2, 9->10) preserves the constraint that a partially +completed wave still ships as a coherent prefix of v1.34.0." (quoted +verbatim from the consultant output, arrows ASCII-normalized) + +## Orchestrator decision + +Variant 1 is adopted without override. It is the same anchor-owned shared +abstraction pattern that succeeded in the v1.32.0 and v1.33.0 waves, applied +to this wave's shape: here the two shared pieces (path-scope engine, +diagnostics-signal model) each ship inside the first consuming feature +commit, keeping every commit release-visible and the layering +one-directional. The accepted ordering constraints (P1 before P2, O2 before +O3) are recorded in `plan.md`. From 7b19e334ecae567dc3893bbe6e2e1285e69d7ab6 Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 20:26:19 +0000 Subject: [PATCH 02/14] fix(cli): treat early-closed stdout pipe as clean exit An early-closed stdout pipe (`o2b | head -1`) surfaces as an EPIPE write failure. That is the normal end of a shell pipeline, not a program error, so the CLI must exit 0 with no diagnostic. A non-EPIPE stdout write failure (ENOSPC, EIO, ...) is a real I/O fault and was being swallowed silently; it now fails loudly with its message and a nonzero exit. The new guard installs a `process.stdout` error listener (Bun surfaces a closed-pipe write as an asynchronous error event) and the entry point's promise catch covers the synchronous-throw case. `vault-log` shares the same `main.ts` entry, so both CLIs are covered. Co-Authored-By: Claude Fable 5 --- src/cli/main.ts | 13 ++++- src/cli/stdout-guard.ts | 68 ++++++++++++++++++++++ tests/cli/stdout-epipe-guard.test.ts | 84 +++++++++++++++++++++++++++ tests/helpers/epipe-stream-harness.ts | 17 ++++++ 4 files changed, 181 insertions(+), 1 deletion(-) create mode 100644 src/cli/stdout-guard.ts create mode 100644 tests/cli/stdout-epipe-guard.test.ts create mode 100644 tests/helpers/epipe-stream-harness.ts diff --git a/src/cli/main.ts b/src/cli/main.ts index 21e631a2..a7655001 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -25,6 +25,7 @@ import { ensureVaultCurrent } from "../core/maintenance/ensure-current.ts"; import { doctor } from "../core/doctor.ts"; import { listVaultPages, writeFrontmatter } from "../core/vault.ts"; import { CliError, parseFlags } from "./argparse.ts"; +import { installStdoutEpipeGuard, isEpipeError } from "./stdout-guard.ts"; import { handleAiderSubcommand } from "./aider.ts"; import { handleBrainSubcommand } from "./brain.ts"; import { handleDisciplineSubcommand } from "./discipline.ts"; @@ -930,5 +931,15 @@ async function dispatchCommand(command: string, rest: string[]): Promise } if (import.meta.main) { - main(process.argv.slice(2)).then((code) => process.exit(code)); + // O1 (t_2ed754d1): an early-closed stdout pipe (`o2b | head -1`) + // must exit clean. The listener catches Bun's asynchronous closed-pipe + // `error` event; the `.catch` covers a synchronous EPIPE throw. Every other + // error keeps its existing loud failure. + installStdoutEpipeGuard(); + main(process.argv.slice(2)) + .then((code) => process.exit(code)) + .catch((err: unknown) => { + if (isEpipeError(err)) process.exit(0); + throw err; + }); } diff --git a/src/cli/stdout-guard.ts b/src/cli/stdout-guard.ts new file mode 100644 index 00000000..913a3c1c --- /dev/null +++ b/src/cli/stdout-guard.ts @@ -0,0 +1,68 @@ +/** + * Stdout EPIPE guard for the CLI entry points (`o2b` and `vault-log`). + * + * When a downstream reader closes the pipe early (`o2b | head -1`), + * the next write to stdout fails with EPIPE. That is the normal, benign end of + * a shell pipeline, not a program error: the CLI must exit 0 with no + * diagnostic. Every OTHER stdout write failure (ENOSPC, EIO, ...) is a real + * I/O fault and must keep failing loudly with its message and a nonzero exit, + * never silently swallowed. + * + * This is the single sanctioned EPIPE special case in the codebase and it is + * scoped to stdout only; error handling elsewhere stays unchanged. + */ + +/** The Node/Bun errno code for a write to a pipe with no reader. */ +export const EPIPE_CODE = "EPIPE"; + +interface ErrnoLike { + readonly code?: unknown; +} + +/** True when `err` is an EPIPE errno error (object carrying `code: "EPIPE"`). */ +export function isEpipeError(err: unknown): boolean { + return typeof err === "object" && err !== null && (err as ErrnoLike).code === EPIPE_CODE; +} + +/** Side-effecting outcome of a stdout error, injected so the mapping is testable. */ +export interface StreamErrorSink { + /** Terminate the process with `code`. */ + readonly exit: (code: number) => never; + /** Emit a diagnostic (to stderr in production). */ + readonly writeError: (message: string) => void; +} + +/** + * Map a stdout stream error to a process outcome: EPIPE exits 0 silently; + * every other error is reported and exits nonzero. Never returns. + */ +export function handleStdoutError(err: unknown, sink: StreamErrorSink): never { + if (isEpipeError(err)) { + return sink.exit(0); + } + const message = err instanceof Error ? err.message : String(err); + sink.writeError(`error: ${message}\n`); + return sink.exit(1); +} + +/** The production sink: real `process.exit` and a fail-soft stderr write. */ +const PROCESS_SINK: StreamErrorSink = { + exit: (code: number): never => process.exit(code), + writeError: (message: string): void => { + try { + process.stderr.write(message); + } catch { + // stderr is gone too; there is nowhere left to report, so just exit. + } + }, +}; + +/** + * Install the guard on `process.stdout` for the real CLI entry point. Bun + * surfaces a closed-pipe write as an asynchronous `error` event rather than a + * synchronous throw, so the listener is the primary path; the entry point's + * promise `.catch` handles the synchronous-throw case (see main.ts). + */ +export function installStdoutEpipeGuard(): void { + process.stdout.on("error", (err: unknown) => handleStdoutError(err, PROCESS_SINK)); +} diff --git a/tests/cli/stdout-epipe-guard.test.ts b/tests/cli/stdout-epipe-guard.test.ts new file mode 100644 index 00000000..31e06143 --- /dev/null +++ b/tests/cli/stdout-epipe-guard.test.ts @@ -0,0 +1,84 @@ +/** + * O1 (t_2ed754d1): an early-closed stdout pipe must exit clean. + * + * Covers the pure mapping (EPIPE -> exit 0, every other stdout error stays + * loud with a nonzero exit) and an end-to-end regression: a real subprocess + * that streams many lines into a pipe a reader closes early must exit 0 with + * no diagnostic on stderr. `vault-log` shares the same `main.ts` entry point, + * so guarding that entry covers both CLIs. + */ + +import { describe, expect, test } from "bun:test"; +import { join } from "node:path"; + +import { handleStdoutError, isEpipeError } from "../../src/cli/stdout-guard.ts"; + +const ROOT = join(import.meta.dir, "..", ".."); + +/** A sink that records the outcome instead of really exiting the process. */ +function recordingSink() { + const errors: string[] = []; + let exitCode: number | null = null; + return { + errors, + get exitCode() { + return exitCode; + }, + sink: { + exit: (code: number): never => { + exitCode = code; + // Stop control flow the way process.exit would. + throw new Error(`__exit_${code}__`); + }, + writeError: (message: string): void => { + errors.push(message); + }, + }, + }; +} + +describe("isEpipeError", () => { + test("recognises an EPIPE errno error object", () => { + expect(isEpipeError({ code: "EPIPE" })).toBe(true); + const err = Object.assign(new Error("write EPIPE"), { code: "EPIPE" }); + expect(isEpipeError(err)).toBe(true); + }); + + test("rejects non-EPIPE errors and non-objects", () => { + expect(isEpipeError({ code: "ENOSPC" })).toBe(false); + expect(isEpipeError(new Error("boom"))).toBe(false); + expect(isEpipeError(null)).toBe(false); + expect(isEpipeError("EPIPE")).toBe(false); + }); +}); + +describe("handleStdoutError", () => { + test("EPIPE maps to a silent exit 0", () => { + const rec = recordingSink(); + expect(() => handleStdoutError({ code: "EPIPE" }, rec.sink)).toThrow("__exit_0__"); + expect(rec.exitCode).toBe(0); + expect(rec.errors).toEqual([]); + }); + + test("a non-EPIPE stdout error fails loud with a nonzero exit", () => { + const rec = recordingSink(); + const err = Object.assign(new Error("no space left on device"), { code: "ENOSPC" }); + expect(() => handleStdoutError(err, rec.sink)).toThrow("__exit_1__"); + expect(rec.exitCode).toBe(1); + expect(rec.errors.join("")).toContain("no space left on device"); + }); +}); + +describe("closed stdout pipe regression", () => { + test("streaming many lines into an early-closed pipe exits 0 with no stderr", async () => { + // Drive the real guard in a fresh process: it writes far more than a pipe + // buffer can hold, and the downstream `head -c1` closes after one byte. + const proc = Bun.spawn( + ["bash", "-c", "bun run tests/helpers/epipe-stream-harness.ts | head -c1 >/dev/null"], + { cwd: ROOT, stdout: "pipe", stderr: "pipe" }, + ); + const [stderr, returncode] = await Promise.all([new Response(proc.stderr).text(), proc.exited]); + expect(stderr).toBe(""); + expect(returncode).toBe(0); + }, 20_000); +}); diff --git a/tests/helpers/epipe-stream-harness.ts b/tests/helpers/epipe-stream-harness.ts new file mode 100644 index 00000000..1af8030b --- /dev/null +++ b/tests/helpers/epipe-stream-harness.ts @@ -0,0 +1,17 @@ +/** + * Test harness for the O1 stdout-EPIPE guard (t_2ed754d1). + * + * Installs the real guard and streams far more than a pipe buffer can hold, so + * a downstream reader that closes early forces a closed-pipe write. Used only + * by `tests/cli/stdout-epipe-guard.test.ts`; it is not part of the shipped CLI. + */ + +import { installStdoutEpipeGuard } from "../../src/cli/stdout-guard.ts"; + +installStdoutEpipeGuard(); + +// ~20 MB across 100k lines: far past any pipe buffer, so a reader that closes +// early guarantees a closed-pipe write, yet small enough to finish promptly. +for (let i = 0; i < 100_000; i++) { + process.stdout.write(`line ${i} ${"x".repeat(200)}\n`); +} From ad09103bf3855830936de5ac7a30673bb19e7787 Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 20:33:34 +0000 Subject: [PATCH 03/14] feat(hygiene): honor nested gitignore files in repo scan The hygiene repo scan only skipped a static list of directory names, so paths ignored by the repo's own .gitignore files were still read and scanned. A new path-scope engine (src/core/fs/ignore.ts) implements the documented gitignore subset: basename vs anchored matching, directory-only patterns, `*`/`?`/doubled-star wildcards, nested composition where a deeper file scopes its subtree, a nearer `!` re-include winning over an outer ignore, and `.git/info/exclude` layered at the lowest precedence. scan-repo composes these per directory during its walk. With no ignore files present the walk is byte-identical to the static skip-dir baseline. Malformed patterns are reported on stderr and produce no rule, so a bad pattern never silently drops a path. The engine sits below both hygiene and ingest so P2 can reuse it without a duplicate matcher. Co-Authored-By: Claude Fable 5 --- src/core/fs/ignore.ts | 243 ++++++++++++++++++++ src/core/hygiene/scan-repo.ts | 100 +++++++- tests/core/fs/ignore.test.ts | 141 ++++++++++++ tests/core/hygiene/scan-repo-ignore.test.ts | 111 +++++++++ 4 files changed, 587 insertions(+), 8 deletions(-) create mode 100644 src/core/fs/ignore.ts create mode 100644 tests/core/fs/ignore.test.ts create mode 100644 tests/core/hygiene/scan-repo-ignore.test.ts diff --git a/src/core/fs/ignore.ts b/src/core/fs/ignore.ts new file mode 100644 index 00000000..8f1ee2b4 --- /dev/null +++ b/src/core/fs/ignore.ts @@ -0,0 +1,243 @@ +/** + * Path-scope engine: a `.gitignore`-style matcher with nested composition. + * + * This is the one home for ignore-pattern matching shared by the hygiene repo + * scan (which honors nested `.gitignore` files and `.git/info/exclude`) and + * source-ingest scoping (which reuses it for `--exclude`). It implements the + * documented gitignore subset - it does NOT claim full `git check-ignore` + * parity. Supported semantics: + * + * - blank lines and `#` comments are skipped; a leading `\#` / `\!` escapes; + * - trailing whitespace is stripped unless backslash-escaped; + * - a `!` prefix negates (re-includes) a match; + * - a leading `/` or an internal `/` anchors the pattern to the file's base + * directory; a slashless pattern matches the basename at any depth; + * - a trailing `/` restricts a pattern to directories; + * - `*` and `?` match within one path segment; a doubled star spans + * segments (trailing, leading, or between two segments); + * - the last matching rule wins, and rules from a deeper file override those + * from a shallower one (nearer-`!`-wins), with `.git/info/exclude` layered + * at the lowest precedence. + * + * A malformed pattern (an unterminated bracket class, or one that fails to + * compile) is reported as a warning and produces no rule - it never silently + * ignores a path. + */ + +/** A pattern that could not be compiled, surfaced instead of silently dropped. */ +export interface IgnoreWarning { + /** Provenance of the ignore file, e.g. `src/.gitignore`. */ + readonly source: string; + /** 1-based line number of the offending pattern. */ + readonly line: number; + /** The raw pattern text. */ + readonly pattern: string; + /** Why it was rejected. */ + readonly reason: string; +} + +interface CompiledRule { + readonly negated: boolean; + readonly dirOnly: boolean; + /** Matches a path expressed relative to the owning layer's base directory. */ + readonly re: RegExp; +} + +/** One parsed ignore file, scoped to the directory it governs. */ +export interface IgnoreLayer { + /** POSIX-relative directory this file governs ("" = repo root). */ + readonly baseDir: string; + readonly rules: readonly CompiledRule[]; +} + +class MalformedPatternError extends Error {} + +/** Escape one literal character for embedding in a regular expression. */ +function escapeLiteral(ch: string): string { + return /[.*+?^${}()|[\]\\/]/.test(ch) ? `\\${ch}` : ch; +} + +/** + * Translate an (already anchor-normalized) glob body into a regex source that + * matches a path relative to its base directory. Throws + * {@link MalformedPatternError} on an unterminated bracket class. + */ +function translateGlob(body: string): string { + let re = ""; + const n = body.length; + let i = 0; + while (i < n) { + const ch = body[i]!; + if (ch === "*") { + const isDouble = body[i + 1] === "*"; + if (isDouble) { + const prevIsBoundary = i === 0 || body[i - 1] === "/"; + let j = i; + while (body[j] === "*") j++; + const nextIsBoundary = j >= n || body[j] === "/"; + if (prevIsBoundary && nextIsBoundary) { + if (body[j] === "/") { + // `**/` - zero or more leading path segments. + re += "(?:[^/]+/)*"; + i = j + 1; + } else { + // trailing `**` - everything below (and the node itself). + re += ".*"; + i = j; + } + } else { + // `**` glued to a segment behaves like a single `*`. + re += "[^/]*"; + i = j; + } + } else { + re += "[^/]*"; + i++; + } + } else if (ch === "?") { + re += "[^/]"; + i++; + } else if (ch === "[") { + let j = i + 1; + if (body[j] === "!" || body[j] === "^") j++; + if (body[j] === "]") j++; // a leading `]` is a literal member + while (j < n && body[j] !== "]") j++; + if (j >= n) throw new MalformedPatternError("unterminated character class"); + let cls = body.slice(i, j + 1); + if (cls.startsWith("[!")) cls = `[^${cls.slice(2)}`; + re += cls; + i = j + 1; + } else { + re += escapeLiteral(ch); + i++; + } + } + return re; +} + +/** Strip trailing whitespace that is not backslash-escaped (git semantics). */ +function stripTrailingWhitespace(line: string): string { + let end = line.length; + while (end > 0 && (line[end - 1] === " " || line[end - 1] === "\t")) { + // A space is retained when the run of preceding backslashes is odd. + let backslashes = 0; + let k = end - 2; + while (k >= 0 && line[k] === "\\") { + backslashes++; + k--; + } + if (backslashes % 2 === 1) break; + end--; + } + return line.slice(0, end); +} + +/** Compile one non-comment, non-blank pattern line into a rule. */ +function compileRule(rawPattern: string): CompiledRule { + let pattern = rawPattern; + const negated = pattern.startsWith("!"); + if (negated) pattern = pattern.slice(1); + else if (pattern.startsWith("\\!") || pattern.startsWith("\\#")) pattern = pattern.slice(1); + + const dirOnly = pattern.endsWith("/"); + if (dirOnly) pattern = pattern.slice(0, -1); + + // Anchored when the pattern carries a leading or internal separator. + const hasLeadingSlash = pattern.startsWith("/"); + if (hasLeadingSlash) pattern = pattern.slice(1); + const anchored = hasLeadingSlash || pattern.includes("/"); + + const translated = translateGlob(pattern); + const source = anchored ? `^${translated}$` : `^(?:.*/)?${translated}$`; + return { negated, dirOnly, re: new RegExp(source) }; +} + +/** + * Parse one ignore file into a layer plus any malformed-pattern warnings. + * `baseDir` is the POSIX-relative directory the file governs ("" = root). + */ +export function parseIgnoreLayer( + content: string, + baseDir: string, + source: string, +): { layer: IgnoreLayer; warnings: IgnoreWarning[] } { + const rules: CompiledRule[] = []; + const warnings: IgnoreWarning[] = []; + const lines = content.split(/\r?\n/); + for (let index = 0; index < lines.length; index++) { + const raw = stripTrailingWhitespace(lines[index]!); + if (raw.length === 0) continue; + if (raw.startsWith("#")) continue; // `\#` was preserved by the strip above + try { + rules.push(compileRule(raw)); + } catch (err) { + warnings.push({ + source, + line: index + 1, + pattern: raw, + reason: err instanceof Error ? err.message : String(err), + }); + } + } + return { layer: { baseDir, rules }, warnings }; +} + +/** Parse a flat list of patterns (e.g. `--exclude` values) into a layer. */ +export function parseIgnorePatterns( + patterns: ReadonlyArray, + baseDir: string, + source: string, +): { layer: IgnoreLayer; warnings: IgnoreWarning[] } { + return parseIgnoreLayer(patterns.join("\n"), baseDir, source); +} + +/** Path relative to `baseDir`, or null when `relPath` is not under it. */ +function relativeToBase(baseDir: string, relPath: string): string | null { + if (baseDir === "") return relPath; + if (relPath === baseDir) return ""; + return relPath.startsWith(`${baseDir}/`) ? relPath.slice(baseDir.length + 1) : null; +} + +/** + * An ordered stack of ignore layers, lowest precedence first. Immutable: + * {@link extend} returns a new scope so a tree walk can push a directory's + * `.gitignore` for its subtree without mutating the parent's scope. + */ +export class IgnoreScope { + private constructor(private readonly layers: readonly IgnoreLayer[]) {} + + static empty(): IgnoreScope { + return new IgnoreScope([]); + } + + /** Return a new scope with `layer` at the highest precedence. */ + extend(layer: IgnoreLayer): IgnoreScope { + return new IgnoreScope([...this.layers, layer]); + } + + /** True when no layer carries any rule (so nothing can be ignored). */ + get isEmpty(): boolean { + return this.layers.every((layer) => layer.rules.length === 0); + } + + /** + * Whether `relPath` (repo-root-relative, POSIX) is ignored. `isDir` gates + * directory-only rules. The last matching rule across all layers in + * precedence order decides; a negated last match re-includes the path. + */ + isIgnored(relPath: string, isDir: boolean): boolean { + let ignored = false; + let matched = false; + for (const layer of this.layers) { + const sub = relativeToBase(layer.baseDir, relPath); + if (sub === null || sub === "") continue; + for (const rule of layer.rules) { + if (rule.dirOnly && !isDir) continue; + if (!rule.re.test(sub)) continue; + matched = true; + ignored = !rule.negated; + } + } + return matched && ignored; + } +} diff --git a/src/core/hygiene/scan-repo.ts b/src/core/hygiene/scan-repo.ts index 5354e6e9..418f811b 100644 --- a/src/core/hygiene/scan-repo.ts +++ b/src/core/hygiene/scan-repo.ts @@ -12,9 +12,10 @@ * are never installed on an operator's machine. */ -import { readdirSync, readFileSync, statSync } from "node:fs"; +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; import { join, relative, sep } from "node:path"; +import { IgnoreScope, parseIgnoreLayer, type IgnoreWarning } from "../fs/ignore.ts"; import { scanFiles, type HardcodedPathFinding } from "./hardcoded-paths.ts"; /** @@ -95,8 +96,69 @@ function isExcludedDir(name: string): boolean { return EXCLUDED_SEGMENTS.has(name); } -/** Recursively collect scannable files under `dir`, fail-soft on I/O. */ -function collectFiles(dir: string, root: string, out: string[]): void { +/** Repo-root-relative POSIX path for an absolute path ("" when equal to root). */ +function toPosixRel(root: string, abs: string): string { + return relative(root, abs).split(sep).join("/"); +} + +/** Read one ignore file into the scope, folding any warnings in. Fail-soft on I/O. */ +function extendWithIgnoreFile( + scope: IgnoreScope, + filePath: string, + baseDir: string, + source: string, + warnings: IgnoreWarning[], +): IgnoreScope { + if (!existsSync(filePath)) return scope; + let content: string; + try { + content = readFileSync(filePath, "utf8"); + } catch { + return scope; // an unreadable ignore file is not a scan failure + } + const parsed = parseIgnoreLayer(content, baseDir, source); + warnings.push(...parsed.warnings); + return scope.extend(parsed.layer); +} + +/** + * Base ignore scope for the whole repo: `.git/info/exclude` at the lowest + * precedence, then the root `.gitignore` above it. Nested `.gitignore` files + * are layered per directory during the walk. + */ +function buildBaseScope(root: string, warnings: IgnoreWarning[]): IgnoreScope { + let scope = IgnoreScope.empty(); + scope = extendWithIgnoreFile( + scope, + join(root, ".git", "info", "exclude"), + "", + ".git/info/exclude", + warnings, + ); + scope = extendWithIgnoreFile(scope, join(root, ".gitignore"), "", ".gitignore", warnings); + return scope; +} + +/** + * Recursively collect scannable files under `dir`, fail-soft on I/O. `scope` + * carries the composed ignore rules from all shallower directories; this + * directory's own `.gitignore` (if any) is layered on top for its subtree, so + * a deeper file scopes only what it governs and a nearer `!` re-include wins. + */ +function collectFiles( + dir: string, + root: string, + scope: IgnoreScope, + out: string[], + warnings: IgnoreWarning[], +): void { + const dirScope = extendWithIgnoreFile( + scope, + join(dir, ".gitignore"), + toPosixRel(root, dir), + `${toPosixRel(root, dir)}/.gitignore`, + warnings, + ); let entries; try { entries = readdirSync(dir, { withFileTypes: true }); @@ -105,26 +167,47 @@ function collectFiles(dir: string, root: string, out: string[]): void { } for (const entry of entries) { const abs = join(dir, entry.name); + const posixRel = toPosixRel(root, abs); if (entry.isDirectory()) { if (isExcludedDir(entry.name)) continue; - collectFiles(abs, root, out); + if (dirScope.isIgnored(posixRel, true)) continue; + collectFiles(abs, root, dirScope, out, warnings); } else if (entry.isFile() && hasScanExtension(entry.name)) { - const rel = relative(root, abs); - if (EXCLUDED_RELPATHS.has(rel)) continue; + if (EXCLUDED_RELPATHS.has(relative(root, abs))) continue; + if (dirScope.isIgnored(posixRel, false)) continue; out.push(abs); } } } +/** Emit one stderr line per malformed ignore pattern, deduplicated. */ +function reportIgnoreWarnings(warnings: ReadonlyArray): void { + const seen = new Set(); + for (const w of warnings) { + const key = `${w.source}:${w.line}`; + if (seen.has(key)) continue; + seen.add(key); + process.stderr.write( + `hygiene: ignoring malformed ignore pattern at ${w.source}:${w.line} ` + + `(${w.pattern}): ${w.reason}\n`, + ); + } +} + /** * Enumerate every in-scope file under `root`, as repo-relative paths. * Deterministic order (sorted after the walk) so reports diff cleanly - * regardless of the readdir ordering the platform returns. + * regardless of the readdir ordering the platform returns. Paths ignored by + * the repo's `.gitignore` files (root and nested) and `.git/info/exclude` are + * skipped in addition to the static skip-dir baseline; with no ignore files + * present the result is byte-identical to the baseline walk. */ export function listScanTargets(root: string): string[] { + const warnings: IgnoreWarning[] = []; + const baseScope = buildBaseScope(root, warnings); const abs: string[] = []; for (const dir of SCAN_DIRS) { - collectFiles(join(root, dir), root, abs); + collectFiles(join(root, dir), root, baseScope, abs, warnings); } for (const name of SCAN_ROOT_FILES) { const p = join(root, name); @@ -134,6 +217,7 @@ export function listScanTargets(root: string): string[] { // absent root file is fine } } + reportIgnoreWarnings(warnings); // Sort to guarantee stable cross-platform output: readdirSync is // alphabetical on POSIX (libuv alphasort) but not guaranteed on Windows. abs.sort((a, b) => (a < b ? -1 : a > b ? 1 : 0)); diff --git a/tests/core/fs/ignore.test.ts b/tests/core/fs/ignore.test.ts new file mode 100644 index 00000000..cb93059d --- /dev/null +++ b/tests/core/fs/ignore.test.ts @@ -0,0 +1,141 @@ +/** + * P1 (t_9654de80): the path-scope engine mirrors the documented gitignore + * subset - basename vs anchored matching, directory-only patterns, `**` + * wildcards, nested composition (deeper file scopes its subtree), nearer-`!` + * re-include winning over an outer ignore, and `.git/info/exclude` layering at + * the lowest precedence. Malformed patterns surface as warnings, never silent + * skips. + */ + +import { describe, expect, test } from "bun:test"; + +import { IgnoreScope, parseIgnoreLayer } from "../../../src/core/fs/ignore.ts"; + +/** Build a scope from `[baseDir, content]` layers, lowest precedence first. */ +function scopeOf(...layers: ReadonlyArray<[string, string]>): IgnoreScope { + let scope = IgnoreScope.empty(); + for (const [baseDir, content] of layers) { + scope = scope.extend( + parseIgnoreLayer(content, baseDir, `${baseDir || ""}/.gitignore`).layer, + ); + } + return scope; +} + +describe("basename vs anchored matching", () => { + test("a slashless pattern matches the basename at any depth", () => { + const scope = scopeOf(["", "*.log\n"]); + expect(scope.isIgnored("a.log", false)).toBe(true); + expect(scope.isIgnored("deep/nested/b.log", false)).toBe(true); + expect(scope.isIgnored("a.logx", false)).toBe(false); + }); + + test("a leading slash anchors to the base directory", () => { + const scope = scopeOf(["", "/build\n"]); + expect(scope.isIgnored("build", true)).toBe(true); + expect(scope.isIgnored("src/build", true)).toBe(false); + }); + + test("an internal slash anchors without a leading slash", () => { + const scope = scopeOf(["", "src/generated\n"]); + expect(scope.isIgnored("src/generated", true)).toBe(true); + expect(scope.isIgnored("pkg/src/generated", true)).toBe(false); + }); +}); + +describe("directory-only patterns", () => { + test("a trailing slash matches directories only", () => { + const scope = scopeOf(["", "dist/\n"]); + expect(scope.isIgnored("dist", true)).toBe(true); + expect(scope.isIgnored("dist", false)).toBe(false); + expect(scope.isIgnored("a/dist", true)).toBe(true); + }); +}); + +describe("wildcards", () => { + test("** matches across directory boundaries", () => { + const scope = scopeOf(["", "logs/**\n"]); + expect(scope.isIgnored("logs/a.txt", false)).toBe(true); + expect(scope.isIgnored("logs/x/y.txt", false)).toBe(true); + }); + + test("a/**/b matches zero or more intermediate directories", () => { + const scope = scopeOf(["", "a/**/b\n"]); + expect(scope.isIgnored("a/b", true)).toBe(true); + expect(scope.isIgnored("a/x/b", true)).toBe(true); + expect(scope.isIgnored("a/x/y/b", true)).toBe(true); + }); + + test("* does not cross a directory boundary", () => { + const scope = scopeOf(["", "a/*.ts\n"]); + expect(scope.isIgnored("a/x.ts", false)).toBe(true); + expect(scope.isIgnored("a/sub/x.ts", false)).toBe(false); + }); +}); + +describe("nested composition", () => { + test("a deeper file scopes only its subtree", () => { + const scope = scopeOf(["", "*.txt\n"], ["sub", "keep-me\n"]); + // The deeper `keep-me` rule only governs paths under sub/. + expect(scope.isIgnored("sub/keep-me", false)).toBe(true); + expect(scope.isIgnored("keep-me", false)).toBe(false); + // The root rule still applies everywhere. + expect(scope.isIgnored("top.txt", false)).toBe(true); + }); + + test("a nearer ! re-include wins over an outer ignore", () => { + const scope = scopeOf(["", "*.env\n"], ["sub", "!prod.env\n"]); + expect(scope.isIgnored("sub/prod.env", false)).toBe(false); + expect(scope.isIgnored("sub/dev.env", false)).toBe(true); + expect(scope.isIgnored("prod.env", false)).toBe(true); + }); +}); + +describe(".git/info/exclude precedence", () => { + test("info/exclude sits below .gitignore, so a root ! re-include wins", () => { + // First layer = info/exclude (lowest), second = root .gitignore (higher). + const scope = scopeOf(["", "generated\n"], ["", "!generated\n"]); + expect(scope.isIgnored("generated", true)).toBe(false); + }); +}); + +describe("comments, blanks, and escapes", () => { + test("comments and blank lines are ignored", () => { + const scope = scopeOf(["", "# a comment\n\n*.tmp\n"]); + expect(scope.isIgnored("x.tmp", false)).toBe(true); + expect(scope.isIgnored("a comment", false)).toBe(false); + }); + + test("a backslash-escaped hash is a literal filename", () => { + const scope = scopeOf(["", "\\#literal\n"]); + expect(scope.isIgnored("#literal", false)).toBe(true); + }); + + test("unescaped trailing whitespace is stripped", () => { + const scope = scopeOf(["", "foo \n"]); + expect(scope.isIgnored("foo", false)).toBe(true); + }); +}); + +describe("malformed patterns", () => { + test("an unterminated character class is warned, not silently applied", () => { + const { layer, warnings } = parseIgnoreLayer("good.txt\n[unterminated\n", "", "x/.gitignore"); + expect(warnings.length).toBe(1); + expect(warnings[0]!.line).toBe(2); + expect(warnings[0]!.source).toBe("x/.gitignore"); + // The valid rule still compiled; the malformed one produced no rule. + const scope = IgnoreScope.empty().extend(layer); + expect(scope.isIgnored("good.txt", false)).toBe(true); + // The malformed pattern must NOT accidentally ignore anything. + expect(scope.isIgnored("[unterminated", false)).toBe(false); + expect(scope.isIgnored("anything", false)).toBe(false); + }); +}); + +describe("empty scope", () => { + test("no layers ignores nothing and reports empty", () => { + const scope = IgnoreScope.empty(); + expect(scope.isEmpty).toBe(true); + expect(scope.isIgnored("anything/at/all", false)).toBe(false); + }); +}); diff --git a/tests/core/hygiene/scan-repo-ignore.test.ts b/tests/core/hygiene/scan-repo-ignore.test.ts new file mode 100644 index 00000000..35d19210 --- /dev/null +++ b/tests/core/hygiene/scan-repo-ignore.test.ts @@ -0,0 +1,111 @@ +/** + * P1 (t_9654de80): the hygiene repo scan honors nested `.gitignore` files and + * `.git/info/exclude`, while staying byte-identical to the static skip-dir + * baseline when a tree carries no ignore files. + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { listScanTargets } from "../../../src/core/hygiene/scan-repo.ts"; + +let root: string; + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "scan-ignore-")); +}); + +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +/** Write `content` to `/`, creating parent directories. */ +function put(rel: string, content = "x\n"): void { + const abs = join(root, rel); + mkdirSync(join(abs, ".."), { recursive: true }); + writeFileSync(abs, content, "utf8"); +} + +describe("byte-identical baseline", () => { + test("with no ignore files, every scannable file is a target", () => { + put("src/a.ts"); + put("src/nested/b.ts"); + put("docs/c.md"); + const targets = listScanTargets(root); + expect(targets).toEqual(["docs/c.md", "src/a.ts", "src/nested/b.ts"]); + }); +}); + +describe("nested gitignore composition", () => { + test("a nested .gitignore scopes only its own subtree", () => { + put("src/keep.ts"); + put("src/pkg/drop.ts"); + put("src/pkg/keep.ts"); + put("src/other/drop.ts"); + // The nested file under src/pkg only governs src/pkg. + put("src/pkg/.gitignore", "drop.ts\n"); + const targets = listScanTargets(root); + expect(targets).toContain("src/keep.ts"); + expect(targets).toContain("src/pkg/keep.ts"); + expect(targets).toContain("src/other/drop.ts"); + expect(targets).not.toContain("src/pkg/drop.ts"); + }); + + test("the root .gitignore skips paths under a scan dir", () => { + put("src/a.ts"); + put("src/generated/b.ts"); + put(".gitignore", "generated/\n"); + const targets = listScanTargets(root); + expect(targets).toContain("src/a.ts"); + expect(targets).not.toContain("src/generated/b.ts"); + }); + + test("a nearer ! re-include wins over an outer ignore", () => { + put("src/skip/important.ts"); + put("src/keep/important.ts"); + put("src/keep/other.ts"); + put(".gitignore", "important.ts\n"); + put("src/keep/.gitignore", "!important.ts\n"); + const targets = listScanTargets(root); + // Outer rule ignores important.ts everywhere... + expect(targets).not.toContain("src/skip/important.ts"); + // ...but the nearer re-include under src/keep wins. + expect(targets).toContain("src/keep/important.ts"); + expect(targets).toContain("src/keep/other.ts"); + }); + + test(".git/info/exclude participates in the scan", () => { + put("src/a.ts"); + put("src/secret.ts"); + put(".git/info/exclude", "secret.ts\n"); + const targets = listScanTargets(root); + expect(targets).toContain("src/a.ts"); + expect(targets).not.toContain("src/secret.ts"); + }); +}); + +describe("malformed patterns", () => { + test("a malformed pattern warns on stderr and never silently skips", () => { + put("src/a.ts"); + put("src/[weird.ts"); + put(".gitignore", "[unterminated\n"); + const lines: string[] = []; + const real = process.stderr.write.bind(process.stderr); + process.stderr.write = ((chunk: unknown) => { + lines.push(typeof chunk === "string" ? chunk : String(chunk)); + return true; + }) as typeof process.stderr.write; + let targets: string[]; + try { + targets = listScanTargets(root); + } finally { + process.stderr.write = real; + } + // Malformed rule produced no matcher, so nothing was silently dropped. + expect(targets).toContain("src/a.ts"); + expect(targets).toContain("src/[weird.ts"); + expect(lines.join("")).toContain("malformed ignore pattern"); + }); +}); From e3fc4bc2a14e84e6125ee36e39174d8db0a12e6b Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 20:38:02 +0000 Subject: [PATCH 04/14] feat(ingest): add --src-subpath and --exclude source scoping Folder-ingest planning could only walk a whole source directory. It now accepts two optional scoping controls, plumbed through the batch-plan CLI verb and the brain_ingest_batch_plan MCP tool: - --src-subpath restricts discovery to a subtree of the source dir; a value escaping the source root rejects with a typed error. - --exclude takes gitignore-style patterns matched by the shared src/core/fs/ignore.ts engine (no duplicate matcher); an excluded directory is pruned so its subtree is never walked. A malformed pattern is a typed error, never a silent skip. With neither flag set the plan (paths, planId, sourceDir) is byte-identical to before. No new MCP tools are added. Co-Authored-By: Claude Fable 5 --- src/cli/brain/verbs/batch-plan.ts | 15 ++- src/core/brain/ingest/batch-plan.ts | 64 ++++++++++++- src/mcp/brain/ingest-tools.ts | 23 ++++- tests/cli/brain-batch-plan-scoping.test.ts | 77 ++++++++++++++++ .../brain/ingest/batch-plan-scoping.test.ts | 92 +++++++++++++++++++ 5 files changed, 263 insertions(+), 8 deletions(-) create mode 100644 tests/cli/brain-batch-plan-scoping.test.ts create mode 100644 tests/core/brain/ingest/batch-plan-scoping.test.ts diff --git a/src/cli/brain/verbs/batch-plan.ts b/src/cli/brain/verbs/batch-plan.ts index 031078cd..07c804cf 100644 --- a/src/cli/brain/verbs/batch-plan.ts +++ b/src/cli/brain/verbs/batch-plan.ts @@ -31,13 +31,16 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise { vault: { type: "string" }, "max-bytes": { type: "string" }, "max-files": { type: "string" }, + "src-subpath": { type: "string" }, + exclude: { type: "string-array" }, resume: { type: "boolean" }, json: { type: "boolean" }, }); const sourceDir = positional[0]; if (!sourceDir) { return usageError( - "usage: o2b brain batch-plan [--max-bytes N] [--max-files N] [--resume] [--json]", + "usage: o2b brain batch-plan [--max-bytes N] [--max-files N] " + + "[--src-subpath PATH] [--exclude PATTERN]... [--resume] [--json]", ); } @@ -46,7 +49,15 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise { const maxBatchBytes = parseCap(flags["max-bytes"], "--max-bytes", DEFAULT_MAX_BATCH_BYTES); const maxBatchFiles = parseCap(flags["max-files"], "--max-files", DEFAULT_MAX_BATCH_FILES); const resume = flags["resume"] === true; - const plan = planBatches(vault, sourceDir, { maxBatchBytes, maxBatchFiles, resume }); + const srcSubpath = flags["src-subpath"] as string | undefined; + const exclude = (flags["exclude"] as string[] | undefined) ?? []; + const plan = planBatches(vault, sourceDir, { + maxBatchBytes, + maxBatchFiles, + resume, + ...(srcSubpath !== undefined ? { srcSubpath } : {}), + ...(exclude.length > 0 ? { exclude } : {}), + }); if (flags["json"]) { okJson({ diff --git a/src/core/brain/ingest/batch-plan.ts b/src/core/brain/ingest/batch-plan.ts index 7b7273c1..56f252cb 100644 --- a/src/core/brain/ingest/batch-plan.ts +++ b/src/core/brain/ingest/batch-plan.ts @@ -26,6 +26,7 @@ import { existsSync, readdirSync, statSync } from "node:fs"; import { join, posix, relative } from "node:path"; +import { IgnoreScope, parseIgnorePatterns } from "../../fs/ignore.ts"; import { canonicalNotePath, ensureInsideVault } from "../../path-safety.ts"; import { computePlanId, readCheckpoint } from "./checkpoint.ts"; import { classifyPaths, readManifest } from "./content-manifest.ts"; @@ -62,6 +63,18 @@ export interface BatchPlanOptions { * set so it is stable across resumes. Absent → a fresh, checkpoint-blind plan. */ readonly resume?: boolean; + /** + * Scope discovery to a subtree of `sourceDir` (t_e82101a5), e.g. `pkg/a` in a + * monorepo. Resolved relative to the source dir; a value escaping it throws. + * Absent → the whole source dir is walked (byte-identical to before). + */ + readonly srcSubpath?: string; + /** + * Gitignore-style patterns (t_e82101a5), matched by the shared + * {@link parseIgnorePatterns} engine relative to `sourceDir`. Discovered + * paths matching any pattern are dropped. Absent/empty → no exclusion. + */ + readonly exclude?: readonly string[]; } /** One file selected for (re-)ingest, with the reason it was selected. */ @@ -138,11 +151,25 @@ export function planBatches(vault: string, sourceDir: string, opts: BatchPlanOpt throw new Error(`planBatches: source dir is not an existing directory: ${sourceDir}`); } + // Optional subtree scoping (t_e82101a5): resolve `srcSubpath` under the + // source dir. ensureInsideVault throws a typed error if it escapes. + let walkRoot = dirAbs; + if (opts.srcSubpath !== undefined && opts.srcSubpath.length > 0) { + walkRoot = ensureInsideVault(join(dirAbs, opts.srcSubpath), dirAbs); + if (!existsSync(walkRoot) || !statSync(walkRoot).isDirectory()) { + throw new Error(`planBatches: src subpath is not an existing directory: ${opts.srcSubpath}`); + } + } + const extensions = normalizeExtensions(opts.extensions ?? DEFAULT_INGESTIBLE_EXTENSIONS); + // Optional exclusion (t_e82101a5): reuse the shared ignore engine so there is + // no second matcher. Patterns are relative to the source dir. A malformed + // pattern is a user error, surfaced loudly rather than silently ignored. + const excludeScope = buildExcludeScope(opts.exclude ?? [], dirRel); // Discover ingestible files as canonical vault-relative paths, sorted. const discovered: string[] = []; - collectIngestible(dirAbs, extensions, discovered); + collectIngestible(walkRoot, vault, extensions, excludeScope, discovered); const relPaths = discovered.map((abs) => canonicalNotePath(toPosixRel(vault, abs))).toSorted(); // The plan id keys on the FULL discovered set, so it is identical before and @@ -225,14 +252,43 @@ function packBatches( return batches; } -/** Recursively collect ingestible regular-file absolute paths, skipping hidden entries. */ -function collectIngestible(dir: string, extensions: ReadonlySet, out: string[]): void { +/** + * Build the exclusion scope from `--exclude` patterns, relative to the source + * dir. A malformed pattern is a user error and throws (never a silent skip). An + * empty pattern list yields an empty scope that ignores nothing, keeping the + * no-flag path byte-identical. + */ +function buildExcludeScope(patterns: readonly string[], baseDir: string): IgnoreScope { + if (patterns.length === 0) return IgnoreScope.empty(); + const { layer, warnings } = parseIgnorePatterns(patterns, baseDir, "--exclude"); + if (warnings.length > 0) { + const w = warnings[0]!; + throw new Error(`planBatches: malformed --exclude pattern "${w.pattern}": ${w.reason}`); + } + return IgnoreScope.empty().extend(layer); +} + +/** + * Recursively collect ingestible regular-file absolute paths, skipping hidden + * entries and anything the exclusion scope matches (t_e82101a5). An excluded + * directory is pruned so its subtree is never walked. + */ +function collectIngestible( + dir: string, + vault: string, + extensions: ReadonlySet, + exclude: IgnoreScope, + out: string[], +): void { for (const entry of readdirSync(dir, { withFileTypes: true })) { if (entry.name.startsWith(".")) continue; // hidden file or dot-directory const abs = join(dir, entry.name); + const rel = toPosixRel(vault, abs); if (entry.isDirectory()) { - collectIngestible(abs, extensions, out); + if (exclude.isIgnored(rel, true)) continue; + collectIngestible(abs, vault, extensions, exclude, out); } else if (entry.isFile() && extensions.has(extname(entry.name))) { + if (exclude.isIgnored(rel, false)) continue; out.push(abs); } } diff --git a/src/mcp/brain/ingest-tools.ts b/src/mcp/brain/ingest-tools.ts index d75bd8d2..092d34f6 100644 --- a/src/mcp/brain/ingest-tools.ts +++ b/src/mcp/brain/ingest-tools.ts @@ -20,7 +20,7 @@ import { type SourceCleanupPlan, } from "../../core/brain/source-cleanup.ts"; import { resolveAgentName } from "../../core/config.ts"; -import { coerceBoolOptional, coerceInt, coerceStr } from "../coerce.ts"; +import { coerceBoolOptional, coerceInt, coerceStr, coerceStrList } from "../coerce.ts"; import { MCP_PREVIEW_BUDGET } from "../preview-budget.ts"; import type { ServerContext, ToolDefinition } from "../tool-contract.ts"; import { parseExtractionIntakeArgs } from "./intake-args.ts"; @@ -191,7 +191,15 @@ async function toolBrainIngestBatchPlan( Number.MAX_SAFE_INTEGER, ); const resume = coerceBoolOptional(args, "resume") ?? false; - const plan = planBatches(ctx.vault, sourceDir, { maxBatchBytes, maxBatchFiles, resume }); + const srcSubpath = coerceStr(args, "src_subpath", false) ?? undefined; + const exclude = coerceStrList(args, "exclude"); + const plan = planBatches(ctx.vault, sourceDir, { + maxBatchBytes, + maxBatchFiles, + resume, + ...(srcSubpath !== undefined ? { srcSubpath } : {}), + ...(exclude.length > 0 ? { exclude } : {}), + }); // A resumed plan that comes back empty is fully drained: drop its checkpoint // (the content manifest is the authoritative final state from here on). if (resume && plan.batches.length === 0) { @@ -346,6 +354,17 @@ export const INGEST_TOOLS: ReadonlyArray = Object.freeze([ description: "Resume an interrupted plan: exclude items already recorded completed in this plan's checkpoint. Default false.", }, + src_subpath: { + type: "string", + description: + "Scope discovery to a subtree of source_dir (e.g. pkg/a). A value escaping source_dir is a typed error. Absent walks the whole dir.", + }, + exclude: { + type: "array", + items: { type: "string" }, + description: + "Gitignore-style patterns (relative to source_dir) whose matches are dropped from discovery. Same engine as the hygiene scan.", + }, }, required: ["source_dir"], additionalProperties: false, diff --git a/tests/cli/brain-batch-plan-scoping.test.ts b/tests/cli/brain-batch-plan-scoping.test.ts new file mode 100644 index 00000000..11a5c1cc --- /dev/null +++ b/tests/cli/brain-batch-plan-scoping.test.ts @@ -0,0 +1,77 @@ +/** + * P2 (t_e82101a5): the `o2b brain batch-plan` CLI plumbs `--src-subpath` and + * `--exclude` through to planBatches. Behavior is covered at the core level; + * this asserts the flags reach the planner and that omitting them is unchanged. + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { runCli } from "../helpers/run-cli.ts"; + +let vault: string; +let configHome: string; +let configPath: string; + +beforeEach(() => { + vault = mkdtempSync(join(tmpdir(), "o2b-scope-cli-vault-")); + configHome = mkdtempSync(join(tmpdir(), "o2b-scope-cli-cfg-")); + configPath = join(configHome, "config.yaml"); + writeFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`, "utf8"); +}); + +afterEach(() => { + rmSync(vault, { recursive: true, force: true }); + rmSync(configHome, { recursive: true, force: true }); +}); + +function write(rel: string): void { + const abs = join(vault, rel); + mkdirSync(join(abs, ".."), { recursive: true }); + writeFileSync(abs, "content\n", "utf8"); +} + +const ENV = () => ({ OPEN_SECOND_BRAIN_CONFIG: configPath }); + +describe("o2b brain batch-plan scoping flags", () => { + test("--src-subpath restricts discovery to the subtree", async () => { + write("mono/pkg/a/one.md"); + write("mono/pkg/b/two.md"); + const res = await runCli(["brain", "batch-plan", "mono", "--src-subpath", "pkg/a", "--json"], { + env: ENV(), + }); + expect(res.returncode).toBe(0); + const plan = JSON.parse(res.stdout); + const paths = plan.batches.flatMap((b: { files: { path: string }[] }) => + b.files.map((f) => f.path), + ); + expect(paths).toContain("mono/pkg/a/one.md"); + expect(paths).not.toContain("mono/pkg/b/two.md"); + }); + + test("--exclude drops matching files", async () => { + write("mono/keep.md"); + write("mono/vendor/dep.md"); + const res = await runCli(["brain", "batch-plan", "mono", "--exclude", "vendor/", "--json"], { + env: ENV(), + }); + expect(res.returncode).toBe(0); + const plan = JSON.parse(res.stdout); + const paths = plan.batches.flatMap((b: { files: { path: string }[] }) => + b.files.map((f) => f.path), + ); + expect(paths).toEqual(["mono/keep.md"]); + }); + + test("a subpath escaping the source root fails with a nonzero exit", async () => { + write("mono/a.md"); + const res = await runCli( + ["brain", "batch-plan", "mono", "--src-subpath", "../../etc", "--json"], + { env: ENV() }, + ); + expect(res.returncode).not.toBe(0); + expect(res.stderr).toMatch(/escapes|outside/i); + }); +}); diff --git a/tests/core/brain/ingest/batch-plan-scoping.test.ts b/tests/core/brain/ingest/batch-plan-scoping.test.ts new file mode 100644 index 00000000..187579e8 --- /dev/null +++ b/tests/core/brain/ingest/batch-plan-scoping.test.ts @@ -0,0 +1,92 @@ +/** + * P2 (t_e82101a5): source-ingest scoping. `planBatches` gains `--src-subpath` + * (restrict discovery to a subtree) and `--exclude` (gitignore-style patterns + * via the shared src/core/fs/ignore.ts engine). Without the flags the plan is + * byte-identical to today; a subpath escaping the source root is a typed error. + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { bootstrapBrain } from "../../../../src/core/brain/init.ts"; +import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts"; +import { planBatches } from "../../../../src/core/brain/ingest/batch-plan.ts"; + +let vault: string; +let configHome: string; + +beforeEach(() => { + vault = mkdtempSync(join(tmpdir(), "o2b-scope-vault-")); + configHome = mkdtempSync(join(tmpdir(), "o2b-scope-cfg-")); + const configPath = join(configHome, "config.yaml"); + atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`); + bootstrapBrain(vault, { configPath }); +}); + +afterEach(() => { + rmSync(vault, { recursive: true, force: true }); + rmSync(configHome, { recursive: true, force: true }); +}); + +function write(rel: string, content = "hello\n"): void { + const abs = join(vault, rel); + mkdirSync(join(abs, ".."), { recursive: true }); + writeFileSync(abs, content, "utf8"); +} + +function plannedPaths(plan: ReturnType): string[] { + return plan.batches.flatMap((b) => b.files.map((f) => f.path)).toSorted(); +} + +const CAPS = { maxBatchBytes: 100_000, maxBatchFiles: 100 } as const; + +describe("--src-subpath", () => { + test("restricts discovery to the named subtree", () => { + write("mono/pkg/a/one.md"); + write("mono/pkg/a/two.md"); + write("mono/pkg/b/three.md"); + const plan = planBatches(vault, "mono", { ...CAPS, srcSubpath: "pkg/a" }); + expect(plannedPaths(plan)).toEqual(["mono/pkg/a/one.md", "mono/pkg/a/two.md"]); + }); + + test("a subpath escaping the source root is a typed error", () => { + write("mono/pkg/a/one.md"); + expect(() => planBatches(vault, "mono", { ...CAPS, srcSubpath: "../../etc" })).toThrow( + /escapes|outside/i, + ); + }); + + test("a subpath that is not an existing directory throws", () => { + write("mono/pkg/a/one.md"); + expect(() => planBatches(vault, "mono", { ...CAPS, srcSubpath: "pkg/missing" })).toThrow(); + }); +}); + +describe("--exclude", () => { + test("excludes files matching the pattern, composing with the ignore engine", () => { + write("mono/keep.md"); + write("mono/vendor/dep.md"); + write("mono/vendor/nested/deep.md"); + write("mono/notes.tmp.md"); + const plan = planBatches(vault, "mono", { + ...CAPS, + exclude: ["vendor/", "*.tmp.md"], + }); + expect(plannedPaths(plan)).toEqual(["mono/keep.md"]); + }); +}); + +describe("byte-identical opt-out", () => { + test("without the new flags the plan matches the pre-flag output exactly", () => { + write("mono/a.md"); + write("mono/sub/b.md"); + const baseline = planBatches(vault, "mono", CAPS); + const withEmpty = planBatches(vault, "mono", { ...CAPS, exclude: [] }); + expect(baseline.planId).toBe(withEmpty.planId); + expect(plannedPaths(baseline)).toEqual(["mono/a.md", "mono/sub/b.md"]); + expect(plannedPaths(withEmpty)).toEqual(plannedPaths(baseline)); + expect(withEmpty.sourceDir).toBe(baseline.sourceDir); + }); +}); From c84b52fcc28b823d9890477b65366addde953651 Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 20:43:37 +0000 Subject: [PATCH 05/14] feat(ingest): honor extractable flag during page discovery The schema pack stored an `extractable` allowlist of schema tokens (set via the set_extractable mutation) but nothing consulted it. Folder-ingest page discovery now gates on it: when the allowlist is non-empty, a discovered page whose `schema_type` is not in it is skipped before extraction, reported on the plan as skipped-with-reason, and logged to stderr rather than silently ingested. A page with no declared type stays ungated, and an empty allowlist gates nothing so the plan (paths, planId, and serialized output) is byte-identical to before. The gate is a pure, read-only helper that reads frontmatter and returns a partition; it touches no schema mutation surface. The batch-plan CLI verb and brain_ingest_batch_plan MCP tool surface the skipped set only when non-empty. Co-Authored-By: Claude Fable 5 --- src/cli/brain/verbs/batch-plan.ts | 14 +++ src/core/brain/ingest/batch-plan.ts | 27 +++++- src/core/brain/ingest/extractable-gate.ts | 87 +++++++++++++++++++ src/mcp/brain/ingest-tools.ts | 10 +++ .../ingest/batch-plan-extractable.test.ts | 85 ++++++++++++++++++ .../brain/ingest/extractable-gate.test.ts | 77 ++++++++++++++++ 6 files changed, 299 insertions(+), 1 deletion(-) create mode 100644 src/core/brain/ingest/extractable-gate.ts create mode 100644 tests/core/brain/ingest/batch-plan-extractable.test.ts create mode 100644 tests/core/brain/ingest/extractable-gate.test.ts diff --git a/src/cli/brain/verbs/batch-plan.ts b/src/cli/brain/verbs/batch-plan.ts index 07c804cf..4a771914 100644 --- a/src/cli/brain/verbs/batch-plan.ts +++ b/src/cli/brain/verbs/batch-plan.ts @@ -67,6 +67,16 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise { total_files: plan.totalFiles, total_bytes: plan.totalBytes, skipped: [...plan.skipped], + // Only present when the extractable gate skipped something, so the + // no-declaration output is byte-identical to before. + ...(plan.skippedNonExtractable.length > 0 + ? { + skipped_non_extractable: plan.skippedNonExtractable.map((s) => ({ + path: s.path, + reason: s.reason, + })), + } + : {}), plan_id: plan.planId, resumed_completed: plan.resumedCompleted, batches: plan.batches.map((b) => ({ @@ -92,6 +102,10 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise { info(` ${plan.skipped.length} unchanged file(s) skipped:`); for (const p of plan.skipped) info(` - ${p}`); } + if (plan.skippedNonExtractable.length > 0) { + info(` ${plan.skippedNonExtractable.length} non-extractable page(s) skipped:`); + for (const s of plan.skippedNonExtractable) info(` - ${s.path} (${s.reason})`); + } return 0; } catch (err) { return fail((err as Error).message ?? String(err)); diff --git a/src/core/brain/ingest/batch-plan.ts b/src/core/brain/ingest/batch-plan.ts index 56f252cb..914a690d 100644 --- a/src/core/brain/ingest/batch-plan.ts +++ b/src/core/brain/ingest/batch-plan.ts @@ -30,6 +30,11 @@ import { IgnoreScope, parseIgnorePatterns } from "../../fs/ignore.ts"; import { canonicalNotePath, ensureInsideVault } from "../../path-safety.ts"; import { computePlanId, readCheckpoint } from "./checkpoint.ts"; import { classifyPaths, readManifest } from "./content-manifest.ts"; +import { + extractableAllowlist, + partitionExtractable, + type SkippedPage, +} from "./extractable-gate.ts"; /** * Default set of ingestible (text-bearing) extensions, lowercase with the dot. @@ -106,6 +111,12 @@ export interface BatchPlan { readonly batches: readonly IngestBatch[]; /** Canonical paths classified `unchanged` and therefore skipped, sorted. */ readonly skipped: readonly string[]; + /** + * Pages skipped before extraction because their `schema_type` is not in the + * schema `extractable` allowlist (t_ed856388). Empty when the allowlist is + * unset (the gate is off), keeping the plan byte-identical to before. + */ + readonly skippedNonExtractable: readonly SkippedPage[]; /** Total number of files across all batches (`new` + `modified`). */ readonly totalFiles: number; /** Total bytes across all batches. */ @@ -170,7 +181,20 @@ export function planBatches(vault: string, sourceDir: string, opts: BatchPlanOpt // Discover ingestible files as canonical vault-relative paths, sorted. const discovered: string[] = []; collectIngestible(walkRoot, vault, extensions, excludeScope, discovered); - const relPaths = discovered.map((abs) => canonicalNotePath(toPosixRel(vault, abs))).toSorted(); + const discoveredRel = discovered + .map((abs) => canonicalNotePath(toPosixRel(vault, abs))) + .toSorted(); + + // Honor the schema `extractable` allowlist (t_ed856388): pages whose + // schema_type is not extractable are skipped BEFORE extraction, reported with + // a reason, and logged. An empty allowlist gates nothing, so the discovered + // set (hence planId and the whole plan) stays byte-identical to before. + const partition = partitionExtractable(vault, discoveredRel, extractableAllowlist(vault)); + const relPaths = partition.extractable; + const skippedNonExtractable = partition.skipped; + for (const s of skippedNonExtractable) { + process.stderr.write(`batch-plan: skipping non-extractable page ${s.path} (${s.reason})\n`); + } // The plan id keys on the FULL discovered set, so it is identical before and // after an interruption regardless of how many items have completed. @@ -210,6 +234,7 @@ export function planBatches(vault: string, sourceDir: string, opts: BatchPlanOpt maxBatchFiles: opts.maxBatchFiles, batches, skipped, + skippedNonExtractable, totalFiles: planned.length, totalBytes: planned.reduce((sum, f) => sum + f.bytes, 0), planId, diff --git a/src/core/brain/ingest/extractable-gate.ts b/src/core/brain/ingest/extractable-gate.ts new file mode 100644 index 00000000..ed37ffa4 --- /dev/null +++ b/src/core/brain/ingest/extractable-gate.ts @@ -0,0 +1,87 @@ +/** + * Extractable-flag gate for page discovery (P3, t_ed856388). + * + * The schema pack already stores an `extractable` allowlist of schema tokens + * (set via the `set_extractable` mutation), but nothing consulted it during + * discovery. This gate does: when the allowlist is non-empty, a discovered page + * whose declared `schema_type` is NOT in it is skipped before extraction and + * reported with a reason, rather than silently ingested. + * + * Semantics (documented judgment calls): + * - an EMPTY allowlist gates nothing, so discovery is byte-identical to + * before the flag was honored (opt-out by default); + * - a page with no `schema_type` frontmatter belongs to no pack and stays + * ungated (kept) - raw untyped sources are never dropped by this gate. + * + * Pure and read-only: it reads frontmatter and returns a partition; it mutates + * no schema surface. + */ + +import { join } from "node:path"; + +import { parseFrontmatter } from "../../vault.ts"; +import { loadSchemaPack } from "../schema-pack.ts"; + +/** The frontmatter field that carries a page's schema page-type token. */ +const PAGE_TYPE_FIELD = "schema_type"; + +/** One page skipped by the gate, with the reason it was excluded. */ +export interface SkippedPage { + readonly path: string; + readonly reason: string; +} + +/** Discovered pages split into the extractable set and the skipped set. */ +export interface ExtractablePartition { + readonly extractable: string[]; + readonly skipped: SkippedPage[]; +} + +/** + * The set of schema tokens declared extractable for `vault`. An empty set means + * the gate is inactive (no `extractable` declaration). + */ +export function extractableAllowlist(vault: string): ReadonlySet { + return new Set(loadSchemaPack(vault).extractable); +} + +/** A page's declared `schema_type` token, or null when it has none. */ +function pageType(vault: string, relPath: string): string | null { + try { + const [meta] = parseFrontmatter(join(vault, relPath)); + const raw = meta[PAGE_TYPE_FIELD]; + return typeof raw === "string" && raw.length > 0 ? raw : null; + } catch { + // An unreadable/parseless page has no declared type; leave it ungated. + return null; + } +} + +/** + * Partition `relPaths` (vault-relative, in their given order) into pages that + * pass the extractable gate and pages skipped-with-reason. An empty `allowlist` + * keeps everything. + */ +export function partitionExtractable( + vault: string, + relPaths: readonly string[], + allowlist: ReadonlySet, +): ExtractablePartition { + if (allowlist.size === 0) { + return { extractable: [...relPaths], skipped: [] }; + } + const extractable: string[] = []; + const skipped: SkippedPage[] = []; + for (const path of relPaths) { + const type = pageType(vault, path); + if (type === null || allowlist.has(type)) { + extractable.push(path); + continue; + } + skipped.push({ + path, + reason: `schema_type "${type}" is not in the schema extractable allowlist`, + }); + } + return { extractable, skipped }; +} diff --git a/src/mcp/brain/ingest-tools.ts b/src/mcp/brain/ingest-tools.ts index 092d34f6..f4127dfa 100644 --- a/src/mcp/brain/ingest-tools.ts +++ b/src/mcp/brain/ingest-tools.ts @@ -161,6 +161,16 @@ function serializeBatchPlan(plan: BatchPlan): Record { total_files: plan.totalFiles, total_bytes: plan.totalBytes, skipped: [...plan.skipped], + // Only emitted when the extractable gate skipped something, so a plan with + // no extractable declaration serializes byte-identically to before. + ...(plan.skippedNonExtractable.length > 0 + ? { + skipped_non_extractable: plan.skippedNonExtractable.map((s) => ({ + path: s.path, + reason: s.reason, + })), + } + : {}), plan_id: plan.planId, resumed_completed: plan.resumedCompleted, batches: plan.batches.map((b) => ({ diff --git a/tests/core/brain/ingest/batch-plan-extractable.test.ts b/tests/core/brain/ingest/batch-plan-extractable.test.ts new file mode 100644 index 00000000..dde22c8d --- /dev/null +++ b/tests/core/brain/ingest/batch-plan-extractable.test.ts @@ -0,0 +1,85 @@ +/** + * P3 (t_ed856388): planBatches honors the schema `extractable` allowlist during + * discovery. Non-extractable pages are skipped-with-reason and reported on the + * plan; with no allowlist the plan is byte-identical to before the gate. + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { bootstrapBrain } from "../../../../src/core/brain/init.ts"; +import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts"; +import { planBatches } from "../../../../src/core/brain/ingest/batch-plan.ts"; + +let vault: string; +let configHome: string; + +beforeEach(() => { + vault = mkdtempSync(join(tmpdir(), "o2b-extract-plan-vault-")); + configHome = mkdtempSync(join(tmpdir(), "o2b-extract-plan-cfg-")); + const configPath = join(configHome, "config.yaml"); + atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`); + bootstrapBrain(vault, { configPath }); +}); + +afterEach(() => { + rmSync(vault, { recursive: true, force: true }); + rmSync(configHome, { recursive: true, force: true }); +}); + +function page(rel: string, schemaType?: string): void { + const abs = join(vault, rel); + mkdirSync(join(abs, ".."), { recursive: true }); + const fm = schemaType === undefined ? "" : `schema_type: ${schemaType}\n`; + writeFileSync(abs, `---\ntitle: ${rel}\n${fm}---\n\nbody text\n`, "utf8"); +} + +/** Point the schema pack at an extractable allowlist. */ +function setExtractable(tokens: string[]): void { + const block = tokens.map((t) => ` - ${t}`).join("\n"); + writeFileSync( + join(vault, "Brain", "_brain.yaml"), + `schema_version: 1\nschema:\n page_types:\n - paper\n - memo\n extractable:\n${block}\n`, + "utf8", + ); +} + +const CAPS = { maxBatchBytes: 100_000, maxBatchFiles: 100 } as const; + +function plannedPaths(plan: ReturnType): string[] { + return plan.batches.flatMap((b) => b.files.map((f) => f.path)).toSorted(); +} + +describe("extractable gate in planBatches", () => { + test("skips non-extractable pages and reports them with a reason", () => { + page("Sources/a.md", "paper"); + page("Sources/b.md", "memo"); + page("Sources/c.md", "paper"); + setExtractable(["paper"]); + + const plan = planBatches(vault, "Sources", CAPS); + expect(plannedPaths(plan)).toEqual(["Sources/a.md", "Sources/c.md"]); + expect(plan.skippedNonExtractable.map((s) => s.path)).toEqual(["Sources/b.md"]); + expect(plan.skippedNonExtractable[0]!.reason).toContain("memo"); + }); + + test("with no allowlist the plan is byte-identical (empty gate report)", () => { + page("Sources/a.md", "paper"); + page("Sources/b.md", "memo"); + // Default bootstrap has no extractable declaration. + const plan = planBatches(vault, "Sources", CAPS); + expect(plan.skippedNonExtractable).toEqual([]); + expect(plannedPaths(plan)).toEqual(["Sources/a.md", "Sources/b.md"]); + }); + + test("no schema mutation surface changes: reading the pack does not write it", () => { + page("Sources/a.md", "memo"); + setExtractable(["paper"]); + const before = readFileSync(join(vault, "Brain", "_brain.yaml"), "utf8"); + planBatches(vault, "Sources", CAPS); + const after = readFileSync(join(vault, "Brain", "_brain.yaml"), "utf8"); + expect(after).toBe(before); + }); +}); diff --git a/tests/core/brain/ingest/extractable-gate.test.ts b/tests/core/brain/ingest/extractable-gate.test.ts new file mode 100644 index 00000000..bcfb475d --- /dev/null +++ b/tests/core/brain/ingest/extractable-gate.test.ts @@ -0,0 +1,77 @@ +/** + * P3 (t_ed856388): honor the schema `extractable` allowlist during page + * discovery. When the schema declares extractable tokens, pages whose + * `schema_type` is not in the allowlist are skipped-with-reason before + * extraction; pages with no declared type stay ungated; an empty allowlist + * gates nothing (byte-identical to today). + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { + extractableAllowlist, + partitionExtractable, +} from "../../../../src/core/brain/ingest/extractable-gate.ts"; + +let vault: string; + +beforeEach(() => { + vault = mkdtempSync(join(tmpdir(), "o2b-extractable-")); +}); + +afterEach(() => { + rmSync(vault, { recursive: true, force: true }); +}); + +/** Write a page with an optional `schema_type` frontmatter field. */ +function page(rel: string, schemaType?: string): string { + const abs = join(vault, rel); + mkdirSync(join(abs, ".."), { recursive: true }); + const fm = schemaType === undefined ? "" : `schema_type: ${schemaType}\n`; + writeFileSync(abs, `---\ntitle: ${rel}\n${fm}---\n\nbody\n`, "utf8"); + return rel; +} + +describe("partitionExtractable", () => { + test("skips pages whose schema_type is not in the allowlist", () => { + const paths = [page("a.md", "paper"), page("b.md", "memo"), page("c.md", "paper")]; + const part = partitionExtractable(vault, paths, new Set(["paper"])); + expect(part.extractable).toEqual(["a.md", "c.md"]); + expect(part.skipped.map((s) => s.path)).toEqual(["b.md"]); + expect(part.skipped[0]!.reason).toContain("memo"); + }); + + test("a page with no declared type stays ungated (kept)", () => { + const paths = [page("untyped.md"), page("typed.md", "memo")]; + const part = partitionExtractable(vault, paths, new Set(["paper"])); + expect(part.extractable).toContain("untyped.md"); + expect(part.skipped.map((s) => s.path)).toEqual(["typed.md"]); + }); + + test("an empty allowlist keeps everything (no gating)", () => { + const paths = [page("a.md", "paper"), page("b.md", "memo")]; + const part = partitionExtractable(vault, paths, new Set()); + expect(part.extractable).toEqual(["a.md", "b.md"]); + expect(part.skipped).toEqual([]); + }); +}); + +describe("extractableAllowlist", () => { + test("is empty for a vault with no extractable schema declaration", () => { + expect(extractableAllowlist(vault).size).toBe(0); + }); + + test("reflects the schema pack's extractable tokens", () => { + mkdirSync(join(vault, "Brain"), { recursive: true }); + writeFileSync( + join(vault, "Brain", "_brain.yaml"), + "schema_version: 1\nschema:\n page_types:\n - paper\n extractable:\n - paper\n", + "utf8", + ); + const allow = extractableAllowlist(vault); + expect(allow.has("paper")).toBe(true); + }); +}); From ba0c6efa05f3c14091f412d8ac91a3d20955ecd0 Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 21:01:24 +0000 Subject: [PATCH 06/14] feat(ingest): deterministic code-structure pre-extraction pass Add a no-LLM, stdlib-only pre-ingest pass (P4, t_ef786747) that turns a code source into JSON entity/edge seeds: classes and functions as entity seeds, imports and inheritance as edge seeds. Structural parsing per language family (TypeScript/JavaScript and Python) via a line grammar, deduped and sorted so the same input always yields byte-identical output. Unsupported extensions are reported as unextracted with a reason, never a fake empty success. Wire the pass into ingestSource behind an opt-in preExtract option that surfaces the seeds on the result for the agent step; with the pass off the ingest and its summary page are byte-identical to before. Expose the pass via a new `o2b brain pre-extract` CLI verb and an optional pre_extract flag on brain_ingest_source (no new MCP tool; tool count stays 102). Co-Authored-By: Claude Fable 5 --- src/cli/brain.ts | 3 + src/cli/brain/verbs/index.ts | 1 + src/cli/brain/verbs/pre-extract.ts | 54 +++++ src/cli/command-manifest.ts | 5 + src/core/brain/ingest/ingest.ts | 35 +++ src/core/brain/ingest/pre-extract.ts | 252 ++++++++++++++++++++ src/mcp/brain/ingest-tools.ts | 16 +- tests/cli/brain-pre-extract.test.ts | 65 +++++ tests/core/brain/ingest/ingest.test.ts | 53 ++++ tests/core/brain/ingest/pre-extract.test.ts | 123 ++++++++++ tests/mcp/ingest-tool.test.ts | 29 +++ 11 files changed, 635 insertions(+), 1 deletion(-) create mode 100644 src/cli/brain/verbs/pre-extract.ts create mode 100644 src/core/brain/ingest/pre-extract.ts create mode 100644 tests/cli/brain-pre-extract.test.ts create mode 100644 tests/core/brain/ingest/pre-extract.test.ts diff --git a/src/cli/brain.ts b/src/cli/brain.ts index c0067499..85423179 100644 --- a/src/cli/brain.ts +++ b/src/cli/brain.ts @@ -81,6 +81,7 @@ import { cmdBrainImportSession, cmdBrainForgetSource, cmdBrainBatchPlan, + cmdBrainPreExtract, cmdBrainDistill, cmdBrainHandoff, cmdBrainIntention, @@ -299,6 +300,8 @@ export async function handleBrainSubcommand(argv: ReadonlyArray): Promis return await cmdBrainForgetSource(rest); case "batch-plan": return await cmdBrainBatchPlan(rest); + case "pre-extract": + return await cmdBrainPreExtract(rest); case "distill": return await cmdBrainDistill(rest); case "links": diff --git a/src/cli/brain/verbs/index.ts b/src/cli/brain/verbs/index.ts index 310ba671..973d8085 100644 --- a/src/cli/brain/verbs/index.ts +++ b/src/cli/brain/verbs/index.ts @@ -68,6 +68,7 @@ export { cmdBrainProject } from "./project.ts"; export { cmdBrainSource } from "./source.ts"; export { cmdBrainForgetSource } from "./forget-source.ts"; export { cmdBrainBatchPlan } from "./batch-plan.ts"; +export { cmdBrainPreExtract } from "./pre-extract.ts"; export { cmdBrainDistill } from "./distill.ts"; export { cmdBrainLinks } from "./links.ts"; export { cmdBrainProfile } from "./profile.ts"; diff --git a/src/cli/brain/verbs/pre-extract.ts b/src/cli/brain/verbs/pre-extract.ts new file mode 100644 index 00000000..2b929e47 --- /dev/null +++ b/src/cli/brain/verbs/pre-extract.ts @@ -0,0 +1,54 @@ +/** + * `o2b brain pre-extract ` (P4 / t_ef786747): run the deterministic, + * no-LLM code-structure pre-extraction pass over one source file and print its + * JSON entity/edge seeds. + * + * This is the agent-facing form of the pre-ingest pass: an agent runs it before + * extracting from a code source so the structural seeds (classes, functions, + * imports, inheritance) are available as pre-extracted facts. Read-only and + * deterministic - the kernel runs no model. An unsupported extension is + * reported as unextracted with a reason, never a fake empty success. + */ + +import { existsSync, readFileSync, statSync } from "node:fs"; + +import { preExtractCodeStructure } from "../../../core/brain/ingest/pre-extract.ts"; +import { fail, info, ok, okJson, parse, usageError } from "../helpers.ts"; + +export async function cmdBrainPreExtract(argv: string[]): Promise { + const { flags, positional } = parse(argv, { + json: { type: "boolean" }, + }); + const file = positional[0]; + if (!file) { + return usageError("usage: o2b brain pre-extract [--json]"); + } + + try { + if (!existsSync(file) || !statSync(file).isFile()) { + return fail(`pre-extract: not a readable file: ${file}`); + } + const result = preExtractCodeStructure(file, readFileSync(file, "utf8")); + + if (flags["json"]) { + okJson({ path: file, ...result }); + return 0; + } + + if (!result.extracted) { + // An honest "unextracted" report is data, not a failure; surface it as a + // warning so the operator sees exactly why nothing was seeded. + info(`pre-extract: ${file} unextracted (${result.reason})`); + return 0; + } + ok( + `pre-extract: ${file} (${result.language}) - ` + + `${result.entities.length} entit(ies), ${result.edges.length} edge(s)`, + ); + for (const e of result.entities) ok(` ${e.kind} ${e.name}`); + for (const e of result.edges) ok(` ${e.kind} ${e.from} -> ${e.to}`); + return 0; + } catch (err) { + return fail((err as Error).message ?? String(err)); + } +} diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.ts index c6f8e668..445359c1 100644 --- a/src/cli/command-manifest.ts +++ b/src/cli/command-manifest.ts @@ -219,6 +219,11 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({ flag("json", "boolean"), ], ), + command( + "pre-extract", + "Extract deterministic code structure (classes/functions/imports/inheritance) from a source file as JSON seeds", + [flag("json", "boolean")], + ), command("links", "Normalize wikilink path format across Brain notes"), command("bridges", "Propose, accept, or dismiss embedding-near bridge links"), command("clusters", "Detect link-graph communities and materialize cluster notes"), diff --git a/src/core/brain/ingest/ingest.ts b/src/core/brain/ingest/ingest.ts index 8c89fb12..06f55da6 100644 --- a/src/core/brain/ingest/ingest.ts +++ b/src/core/brain/ingest/ingest.ts @@ -40,6 +40,7 @@ import { } from "../provenance/provenance.ts"; import { recordCompleted } from "./checkpoint.ts"; import { updateManifest } from "./content-manifest.ts"; +import { preExtractCodeStructure, type PreExtractResult } from "./pre-extract.ts"; /** Frontmatter `kind:` marker of an ingested source summary page. */ export const BRAIN_SOURCE_KIND = "brain-source"; @@ -63,6 +64,15 @@ export interface IngestSourceOptions { * state; the checkpoint only tracks plan progress. Absent → no checkpoint. */ readonly planId?: string; + /** + * Run the deterministic code-structure pre-extraction pass (P4, t_ef786747) + * over the source file and surface its JSON seeds on the result, so the agent + * step can use them as pre-extracted facts. Off by default: with the pass + * unset the ingest is byte-identical to before (no file read, no result + * field). The seeds are NEVER written into the summary page - they travel on + * the result only, keeping the persisted page unchanged. + */ + readonly preExtract?: boolean; } export interface IngestSourceResult { @@ -76,6 +86,13 @@ export interface IngestSourceResult { readonly entitiesUpdated: readonly string[]; /** Pre-existing entity ids this source connected to (its connections). */ readonly connections: readonly string[]; + /** + * Code-structure pre-extraction seeds (P4), present only when the pass was + * requested via {@link IngestSourceOptions.preExtract}. `extracted: false` + * when the source is not a supported code file or has no readable bytes - + * never a fake empty success. Absent entirely when the pass was off. + */ + readonly preExtract?: PreExtractResult; } function renderLinkSection(heading: string, ids: readonly string[]): string { @@ -94,6 +111,7 @@ export function ingestSource( opts: IngestSourceOptions, ): IngestSourceResult { const canonicalSource = canonicalNotePath(input.sourcePath); + const preExtract = opts.preExtract === true ? runPreExtract(vault, canonicalSource) : undefined; const sourceLink = `[[${canonicalSource}]]`; const provenance: Provenance = { level: "stated", sources: [sourceLink], premises: [] }; @@ -172,9 +190,26 @@ export function ingestSource( entitiesCreated: intake.entitiesCreated, entitiesUpdated: intake.entitiesUpdated, connections, + ...(preExtract !== undefined ? { preExtract } : {}), }; } +/** + * Run the code-structure pre-extraction pass over a vault-file source. A source + * with no readable file bytes (a URL or identity-only source) cannot be parsed, + * so it is reported as unextracted rather than a fake empty success. + */ +function runPreExtract(vault: string, canonicalSource: string): PreExtractResult { + const abs = join(vault, canonicalSource); + if (!existsSync(abs)) { + return { + extracted: false, + reason: `source has no readable file bytes for code-structure pre-extraction: ${canonicalSource}`, + }; + } + return preExtractCodeStructure(canonicalSource, readFileSync(abs, "utf8")); +} + /** Read a stable `created_at` from an existing summary page, else fall back. */ function readCreatedAt(absPath: string, fallback: string): string { const [meta] = parseFrontmatter(absPath); diff --git a/src/core/brain/ingest/pre-extract.ts b/src/core/brain/ingest/pre-extract.ts new file mode 100644 index 00000000..c503e8b0 --- /dev/null +++ b/src/core/brain/ingest/pre-extract.ts @@ -0,0 +1,252 @@ +/** + * Deterministic, no-LLM code-structure pre-extractor (P4, t_ef786747). + * + * A pre-ingest pass that turns one CODE source into JSON seeds an agent can + * treat as pre-extracted facts: classes and functions become entity seeds, + * imports and inheritance become edge seeds. It runs no model - structural + * parsing per language family via a line grammar - and is a deliberate FALLBACK + * pre-pass, not a codegraph substitute. + * + * Determinism: the output depends only on (path, content). Seeds are deduped + * and sorted with a fixed key, so the same input always yields byte-identical + * JSON. No timestamps, no randomness, no natural-language word lists (only + * programming-language keywords, which are grammar). + * + * Honesty: an unsupported file extension yields an explicit `extracted: false` + * report with a reason - never a fake empty success that would masquerade an + * un-parsed source as "no structure found". A supported language that genuinely + * has no declarations returns an empty-but-`extracted: true` result. + */ + +/** A class/function declaration surfaced as an entity seed. */ +export interface CodeEntitySeed { + readonly kind: "class" | "function"; + readonly name: string; +} + +/** + * A structural relationship surfaced as an edge seed. `imports` runs from the + * source path to a module specifier; `inherits` runs from a subclass to a base + * class (TS `extends`/`implements`, Python base classes). + */ +export interface CodeEdgeSeed { + readonly kind: "imports" | "inherits"; + readonly from: string; + readonly to: string; +} + +/** A source whose language family was recognized and parsed. */ +export interface PreExtractSuccess { + readonly extracted: true; + /** Recognized language family: `typescript`, `javascript`, or `python`. */ + readonly language: string; + /** Class/function seeds, deduped and sorted by (kind, name). */ + readonly entities: readonly CodeEntitySeed[]; + /** Import/inheritance seeds, deduped and sorted by (kind, from, to). */ + readonly edges: readonly CodeEdgeSeed[]; +} + +/** A source whose extension is outside the extractor's supported languages. */ +export interface PreExtractUnsupported { + readonly extracted: false; + readonly reason: string; +} + +export type PreExtractResult = PreExtractSuccess | PreExtractUnsupported; + +/** Recognized language family for a lowercase, dot-prefixed extension. */ +type Language = "typescript" | "javascript" | "python"; + +/** Extension -> language family. The single home for supported extensions. */ +const LANGUAGE_BY_EXTENSION: ReadonlyMap = new Map([ + [".ts", "typescript"], + [".tsx", "typescript"], + [".mts", "typescript"], + [".cts", "typescript"], + [".js", "javascript"], + [".jsx", "javascript"], + [".mjs", "javascript"], + [".cjs", "javascript"], + [".py", "python"], + [".pyi", "python"], +]); + +/** TS/JS class declaration head, capturing the class name. */ +const TS_CLASS = /^(?:export\s+)?(?:default\s+)?(?:abstract\s+)?class\s+([A-Za-z_$][\w$]*)/; +/** TS/JS function declaration head, capturing the function name. */ +const TS_FUNCTION = + /^(?:export\s+)?(?:default\s+)?(?:async\s+)?function\s*\*?\s*([A-Za-z_$][\w$]*)/; +/** `extends ` clause on a TS/JS class line. */ +const TS_EXTENDS = /\bextends\s+([A-Za-z_$][\w$.]*)/; +/** `implements , ` clause on a TS/JS class line (comma list). */ +const TS_IMPLEMENTS = /\bimplements\s+([A-Za-z_$][\w$.,\s]*?)\s*\{/; +/** `import ... from "mod"` / `export ... from "mod"` module specifier. */ +const TS_FROM = /^(?:import|export)\b.*?\bfrom\s*['"]([^'"]+)['"]/; +/** Bare side-effect import `import "mod"`. */ +const TS_BARE_IMPORT = /^import\s*['"]([^'"]+)['"]/; +/** CommonJS `require("mod")` anywhere on the line. */ +const TS_REQUIRE = /\brequire\(\s*['"]([^'"]+)['"]\s*\)/; + +/** Python class head, capturing the name and an optional base-list group. */ +const PY_CLASS = /^class\s+([A-Za-z_]\w*)\s*(?:\(([^)]*)\))?\s*:/; +/** Python function/method head (module-level or indented), capturing the name. */ +const PY_FUNCTION = /^(?:async\s+)?def\s+([A-Za-z_]\w*)\s*\(/; +/** Python `import a, b.c as d` statement body (after the keyword). */ +const PY_IMPORT = /^import\s+(.+?)\s*$/; +/** Python `from import ...`, capturing the (possibly relative) module. */ +const PY_FROM = /^from\s+(\.*[\w.]*)\s+import\b/; +/** A bare Python identifier or dotted name (used to filter base-class args). */ +const PY_DOTTED_NAME = /^[A-Za-z_][\w.]*$/; + +/** + * Extract code structure from `content` addressed by `path`. Returns + * `extracted: false` with a reason when the extension is unsupported, otherwise + * the deduped, sorted entity/edge seeds for the recognized language. + */ +export function preExtractCodeStructure(path: string, content: string): PreExtractResult { + const ext = extensionOf(path); + const language = ext ? LANGUAGE_BY_EXTENSION.get(ext) : undefined; + if (language === undefined) { + const shown = ext.length > 0 ? ext : "(none)"; + return { + extracted: false, + reason: `unsupported source extension "${shown}" for code-structure pre-extraction`, + }; + } + + const entities: CodeEntitySeed[] = []; + const edges: CodeEdgeSeed[] = []; + if (language === "python") { + parsePython(path, content, entities, edges); + } else { + parseTsJs(path, content, entities, edges); + } + + return { + extracted: true, + language, + entities: dedupeSorted(entities, entityKey), + edges: dedupeSorted(edges, edgeKey), + }; +} + +/** Parse the TS/JS line grammar into entity/edge seeds. */ +function parseTsJs( + path: string, + content: string, + entities: CodeEntitySeed[], + edges: CodeEdgeSeed[], +): void { + for (const raw of content.split("\n")) { + const line = raw.trim(); + if (line.length === 0 || isTsComment(line)) continue; + + const classMatch = TS_CLASS.exec(line); + if (classMatch) { + const name = classMatch[1]!; + entities.push({ kind: "class", name }); + const ext = TS_EXTENDS.exec(line); + if (ext) edges.push({ kind: "inherits", from: name, to: ext[1]! }); + const impl = TS_IMPLEMENTS.exec(line); + if (impl) { + for (const base of splitNames(impl[1]!)) { + edges.push({ kind: "inherits", from: name, to: base }); + } + } + } + + const fnMatch = TS_FUNCTION.exec(line); + if (fnMatch) entities.push({ kind: "function", name: fnMatch[1]! }); + + const from = TS_FROM.exec(line) ?? TS_BARE_IMPORT.exec(line); + if (from) edges.push({ kind: "imports", from: path, to: from[1]! }); + const req = TS_REQUIRE.exec(line); + if (req) edges.push({ kind: "imports", from: path, to: req[1]! }); + } +} + +/** Parse the Python line grammar into entity/edge seeds. */ +function parsePython( + path: string, + content: string, + entities: CodeEntitySeed[], + edges: CodeEdgeSeed[], +): void { + for (const raw of content.split("\n")) { + const line = raw.trim(); + if (line.length === 0 || line.startsWith("#")) continue; + + const classMatch = PY_CLASS.exec(line); + if (classMatch) { + const name = classMatch[1]!; + entities.push({ kind: "class", name }); + const bases = classMatch[2]; + if (bases !== undefined) { + for (const base of splitNames(bases)) { + // Keep only real base classes: drop keyword args (metaclass=..., etc) + // and anything that is not a bare or dotted identifier. + if (PY_DOTTED_NAME.test(base)) edges.push({ kind: "inherits", from: name, to: base }); + } + } + continue; + } + + const fnMatch = PY_FUNCTION.exec(line); + if (fnMatch) { + entities.push({ kind: "function", name: fnMatch[1]! }); + continue; + } + + const fromMatch = PY_FROM.exec(line); + if (fromMatch) { + const mod = fromMatch[1]!; + if (mod.length > 0) edges.push({ kind: "imports", from: path, to: mod }); + continue; + } + const importMatch = PY_IMPORT.exec(line); + if (importMatch) { + for (const spec of splitNames(importMatch[1]!)) { + const mod = spec.split(/\s+as\s+/)[0]!.trim(); + if (mod.length > 0) edges.push({ kind: "imports", from: path, to: mod }); + } + } + } +} + +/** Whether a TS/JS line is a comment (line, block-open, or JSDoc continuation). */ +function isTsComment(line: string): boolean { + return line.startsWith("//") || line.startsWith("/*") || line.startsWith("*"); +} + +/** Split a comma-separated name clause into trimmed, non-empty tokens. */ +function splitNames(clause: string): string[] { + return clause + .split(",") + .map((s) => s.trim()) + .filter((s) => s.length > 0); +} + +/** Lowercase, dot-prefixed extension of a path, or "" when it has none. */ +function extensionOf(path: string): string { + const base = path.slice(path.replace(/\\/g, "/").lastIndexOf("/") + 1); + const dot = base.lastIndexOf("."); + return dot <= 0 ? "" : base.slice(dot).toLowerCase(); +} + +function entityKey(e: CodeEntitySeed): string { + return `${e.kind}${e.name}`; +} + +function edgeKey(e: CodeEdgeSeed): string { + return `${e.kind}${e.from}${e.to}`; +} + +/** Dedupe by a stable key and sort by that key, so output is deterministic. */ +function dedupeSorted(items: readonly T[], key: (item: T) => string): T[] { + const byKey = new Map(); + for (const item of items) { + const k = key(item); + if (!byKey.has(k)) byKey.set(k, item); + } + return [...byKey.keys()].toSorted().map((k) => byKey.get(k)!); +} diff --git a/src/mcp/brain/ingest-tools.ts b/src/mcp/brain/ingest-tools.ts index f4127dfa..f5311660 100644 --- a/src/mcp/brain/ingest-tools.ts +++ b/src/mcp/brain/ingest-tools.ts @@ -42,6 +42,7 @@ async function toolBrainIngestSource( const sourcePath = coerceStr(args, "source_path", true)!; const summary = coerceStr(args, "summary", true)!; const planId = coerceStr(args, "plan_id", false) ?? undefined; + const preExtract = coerceBoolOptional(args, "pre_extract") ?? false; const parsed = parseExtractionIntakeArgs(args, TOOL); const agent = parsed.agent && parsed.agent.trim().length > 0 @@ -52,7 +53,12 @@ async function toolBrainIngestSource( const res = ingestSource( ctx.vault, { sourcePath, summary, extraction: parsed.intake }, - { agent, now: new Date(), ...(planId !== undefined ? { planId } : {}) }, + { + agent, + now: new Date(), + ...(planId !== undefined ? { planId } : {}), + ...(preExtract ? { preExtract: true } : {}), + }, ); return { summary_path: res.summaryPath, @@ -60,6 +66,9 @@ async function toolBrainIngestSource( entities_created: [...res.entitiesCreated], entities_updated: [...res.entitiesUpdated], connections: [...res.connections], + // Only emitted when the pre-extract pass ran, so a call without it is + // byte-identical to before (P4). + ...(res.preExtract !== undefined ? { pre_extract: res.preExtract } : {}), }; }); } @@ -274,6 +283,11 @@ export const INGEST_TOOLS: ReadonlyArray = Object.freeze([ description: "Optional batch-plan id (from brain_ingest_batch_plan). When set, a successful ingest records this source into that plan's resume checkpoint.", }, + pre_extract: { + type: "boolean", + description: + "Run the deterministic no-LLM code-structure pass over the source file and return its class/function/import/inheritance seeds under `pre_extract`. Off by default; absent leaves the response unchanged.", + }, }, required: ["source_path", "summary", "entities"], additionalProperties: false, diff --git a/tests/cli/brain-pre-extract.test.ts b/tests/cli/brain-pre-extract.test.ts new file mode 100644 index 00000000..c8ef4f55 --- /dev/null +++ b/tests/cli/brain-pre-extract.test.ts @@ -0,0 +1,65 @@ +/** + * P4 (t_ef786747): the `o2b brain pre-extract` CLI runs the deterministic + * no-LLM code-structure pass over one source file and prints its JSON seeds. + * The extraction itself is covered at the core level; this asserts the verb is + * wired, emits deterministic JSON, and reports unknown languages as unextracted. + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { runCli } from "../helpers/run-cli.ts"; + +let work: string; + +beforeEach(() => { + work = mkdtempSync(join(tmpdir(), "o2b-pre-extract-cli-")); +}); + +afterEach(() => { + rmSync(work, { recursive: true, force: true }); +}); + +describe("o2b brain pre-extract", () => { + test("emits deterministic JSON seeds for a code source", async () => { + const file = join(work, "widget.ts"); + writeFileSync( + file, + 'import { h } from "./dom";\nexport class Widget extends Base {}\n', + "utf8", + ); + const res = await runCli(["brain", "pre-extract", file, "--json"]); + expect(res.returncode).toBe(0); + const out = JSON.parse(res.stdout); + expect(out.extracted).toBe(true); + expect(out.language).toBe("typescript"); + expect(out.entities).toEqual([{ kind: "class", name: "Widget" }]); + expect(out.edges).toEqual([ + { kind: "imports", from: file, to: "./dom" }, + { kind: "inherits", from: "Widget", to: "Base" }, + ]); + }); + + test("reports an unsupported extension as unextracted rather than empty success", async () => { + const file = join(work, "notes.txt"); + writeFileSync(file, "class NotCode {}\n", "utf8"); + const res = await runCli(["brain", "pre-extract", file, "--json"]); + expect(res.returncode).toBe(0); + const out = JSON.parse(res.stdout); + expect(out.extracted).toBe(false); + expect(out.reason).toContain(".txt"); + }); + + test("a missing file fails with a nonzero exit", async () => { + const res = await runCli(["brain", "pre-extract", join(work, "nope.ts"), "--json"]); + expect(res.returncode).not.toBe(0); + }); + + test("without a file argument prints usage and exits nonzero", async () => { + const res = await runCli(["brain", "pre-extract"]); + expect(res.returncode).not.toBe(0); + expect(res.stderr).toContain("usage:"); + }); +}); diff --git a/tests/core/brain/ingest/ingest.test.ts b/tests/core/brain/ingest/ingest.test.ts index 6c9595bc..237da7a9 100644 --- a/tests/core/brain/ingest/ingest.test.ts +++ b/tests/core/brain/ingest/ingest.test.ts @@ -122,3 +122,56 @@ describe("ingestSource", () => { expect(readCheckpoint(vault, planId)).toBeNull(); }); }); + +describe("ingestSource pre-extract pass (P4, t_ef786747)", () => { + const CODE_INPUT = { + sourcePath: "Code/widget.ts", + summary: "A widget module.", + extraction: { entities: [{ category: "concept", name: "Widget" }], relations: [] }, + }; + + function writeCode(): void { + mkdirSync(join(vault, "Code"), { recursive: true }); + writeFileSync( + join(vault, "Code", "widget.ts"), + 'import { h } from "./dom";\nexport class Widget extends Base {}\n', + "utf8", + ); + } + + test("returns deterministic code-structure seeds when the pass is on", () => { + writeCode(); + const res = ingestSource(vault, CODE_INPUT, { agent: "claude", now: NOW, preExtract: true }); + expect(res.preExtract?.extracted).toBe(true); + if (res.preExtract?.extracted) { + expect(res.preExtract.language).toBe("typescript"); + expect(res.preExtract.entities).toEqual([{ kind: "class", name: "Widget" }]); + expect(res.preExtract.edges).toEqual([ + { kind: "imports", from: "Code/widget.ts", to: "./dom" }, + { kind: "inherits", from: "Widget", to: "Base" }, + ]); + } + }); + + test("with the pass off the result carries no seeds and the page is byte-identical", () => { + writeCode(); + // First ingest creates the entity + page; a second (idempotent) re-ingest + // reaches a stable state where the entity already exists, so the page no + // longer changes between runs. Compare that stable off-page against an + // on-page: the only variable left is the pass, proving it never leaks. + ingestSource(vault, CODE_INPUT, { agent: "claude", now: NOW }); + const off = ingestSource(vault, CODE_INPUT, { agent: "claude", now: NOW }); + const offBytes = readSummary(off.summaryPath); + expect(off.preExtract).toBeUndefined(); + + const on = ingestSource(vault, CODE_INPUT, { agent: "claude", now: NOW, preExtract: true }); + expect(readSummary(on.summaryPath)).toBe(offBytes); + expect(on.preExtract?.extracted).toBe(true); + }); + + test("a non-code source reports unextracted rather than a fake empty success", () => { + // INPUT's source is not materialized on disk, so there is nothing to read. + const res = ingestSource(vault, INPUT, { agent: "claude", now: NOW, preExtract: true }); + expect(res.preExtract?.extracted).toBe(false); + }); +}); diff --git a/tests/core/brain/ingest/pre-extract.test.ts b/tests/core/brain/ingest/pre-extract.test.ts new file mode 100644 index 00000000..2be7d162 --- /dev/null +++ b/tests/core/brain/ingest/pre-extract.test.ts @@ -0,0 +1,123 @@ +/** + * P4 (t_ef786747): deterministic, stdlib-only code-structure pre-extractor. + * + * Turns a code source into JSON entity/edge seeds (classes/functions as + * entities; imports and inheritance as edges) without any model. Same input + * yields the same output; unknown languages are reported as unextracted, never + * a fake empty success. Structural parsing only, no natural-language word list. + */ + +import { describe, expect, test } from "bun:test"; + +import { + preExtractCodeStructure, + type PreExtractResult, + type PreExtractSuccess, +} from "../../../../src/core/brain/ingest/pre-extract.ts"; + +function asSuccess(res: PreExtractResult): PreExtractSuccess { + if (!res.extracted) throw new Error(`expected extracted, got: ${res.reason}`); + return res; +} + +const TS_SOURCE = [ + "// leading comment mentioning class Ghost should be ignored", + 'import { readFileSync } from "node:fs";', + 'import { join } from "node:path";', + "export class Animal {}", + "export abstract class Dog extends Animal implements Pet, Runner {}", + "export function makeDog() {}", + "async function helper() {}", + 'const load = require("./loader");', + "", +].join("\n"); + +const PY_SOURCE = [ + "# leading comment mentioning class Ghost should be ignored", + "import os", + "from collections import OrderedDict", + "class Base:", + " def method(self):", + " pass", + "class Derived(Base, metaclass=Meta):", + " pass", + "def top():", + " pass", + "", +].join("\n"); + +describe("preExtractCodeStructure - TypeScript/JavaScript", () => { + test("extracts classes and functions as sorted entity seeds", () => { + const res = asSuccess(preExtractCodeStructure("pkg/a.ts", TS_SOURCE)); + expect(res.language).toBe("typescript"); + expect(res.entities).toEqual([ + { kind: "class", name: "Animal" }, + { kind: "class", name: "Dog" }, + { kind: "function", name: "helper" }, + { kind: "function", name: "makeDog" }, + ]); + }); + + test("extracts import and inheritance edges", () => { + const res = asSuccess(preExtractCodeStructure("pkg/a.ts", TS_SOURCE)); + expect(res.edges).toEqual([ + { kind: "imports", from: "pkg/a.ts", to: "./loader" }, + { kind: "imports", from: "pkg/a.ts", to: "node:fs" }, + { kind: "imports", from: "pkg/a.ts", to: "node:path" }, + { kind: "inherits", from: "Dog", to: "Animal" }, + { kind: "inherits", from: "Dog", to: "Pet" }, + { kind: "inherits", from: "Dog", to: "Runner" }, + ]); + }); + + test("javascript extension reports the javascript family", () => { + const res = asSuccess(preExtractCodeStructure("pkg/a.js", "export function f() {}\n")); + expect(res.language).toBe("javascript"); + expect(res.entities).toEqual([{ kind: "function", name: "f" }]); + }); +}); + +describe("preExtractCodeStructure - Python", () => { + test("extracts classes, functions, imports and base-class edges", () => { + const res = asSuccess(preExtractCodeStructure("pkg/a.py", PY_SOURCE)); + expect(res.language).toBe("python"); + expect(res.entities).toEqual([ + { kind: "class", name: "Base" }, + { kind: "class", name: "Derived" }, + { kind: "function", name: "method" }, + { kind: "function", name: "top" }, + ]); + expect(res.edges).toEqual([ + { kind: "imports", from: "pkg/a.py", to: "collections" }, + { kind: "imports", from: "pkg/a.py", to: "os" }, + { kind: "inherits", from: "Derived", to: "Base" }, + ]); + }); +}); + +describe("preExtractCodeStructure - determinism", () => { + test("same input yields byte-identical JSON output", () => { + const a = preExtractCodeStructure("pkg/a.ts", TS_SOURCE); + const b = preExtractCodeStructure("pkg/a.ts", TS_SOURCE); + expect(JSON.stringify(a)).toBe(JSON.stringify(b)); + }); +}); + +describe("preExtractCodeStructure - unknown languages", () => { + test("an unsupported extension is reported as unextracted, not empty success", () => { + const res = preExtractCodeStructure("notes/plan.txt", "class NotCode {}\n"); + expect(res.extracted).toBe(false); + if (!res.extracted) expect(res.reason).toContain(".txt"); + }); + + test("a path without an extension is reported as unextracted", () => { + const res = preExtractCodeStructure("Makefile", "all:\n\techo hi\n"); + expect(res.extracted).toBe(false); + }); + + test("a known language with no declarations is an honest empty success", () => { + const res = asSuccess(preExtractCodeStructure("pkg/empty.ts", "const x = 1;\n")); + expect(res.entities).toEqual([]); + expect(res.edges).toEqual([]); + }); +}); diff --git a/tests/mcp/ingest-tool.test.ts b/tests/mcp/ingest-tool.test.ts index f937bb58..6a293031 100644 --- a/tests/mcp/ingest-tool.test.ts +++ b/tests/mcp/ingest-tool.test.ts @@ -78,6 +78,35 @@ describe("brain_ingest_source", () => { handler(ctx, { summary: "x", entities: [{ category: "concept", name: "A" }] }), ).rejects.toThrow(MCPError); }); + + test("pre_extract surfaces deterministic code-structure seeds (P4)", async () => { + mkdirSync(join(vault, "Code"), { recursive: true }); + writeFileSync( + join(vault, "Code", "widget.ts"), + 'import { h } from "./dom";\nexport class Widget extends Base {}\n', + "utf8", + ); + const res = (await handler(ctx, { + source_path: "Code/widget.ts", + summary: "A widget.", + entities: [{ category: "concept", name: "Widget" }], + pre_extract: true, + })) as Record; + expect(res["pre_extract"]).toMatchObject({ + extracted: true, + language: "typescript", + entities: [{ kind: "class", name: "Widget" }], + }); + }); + + test("without pre_extract the response omits the seeds field (byte-identical)", async () => { + const res = (await handler(ctx, { + source_path: "Articles/eth.md", + summary: "x", + entities: [{ category: "concept", name: "A" }], + })) as Record; + expect(res["pre_extract"]).toBeUndefined(); + }); }); describe("brain_ingest_batch_plan resume (t_ba1fa5f6)", () => { From 567a7cd14e4271f793facceb7deb290dca7bd586 Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 21:16:14 +0000 Subject: [PATCH 07/14] feat(ingest): reconcile dispatched vs ingested sources per plan Add reconcilePlan (P5, t_d067a153): after a batch plan drains, diff the plan's dispatched set (batch files plus manifest-unchanged skips) against the checkpoint's completed entries and report the gap - each source that was dispatched but never recorded as ingested. It is a warning surface, not a retry: read-only over checkpoint state, idempotent, and it re-dispatches nothing. A plan that dispatched nothing, or whose sources all completed, reports an empty gap explicitly. Surface the report via a `reconcile` flag on brain_ingest_batch_plan and `--reconcile` on `o2b brain batch-plan`; the report rides on the output only when requested, so the default output stays byte-identical. The MCP reconcile runs before any resume-drain checkpoint clear so it reads a live checkpoint. No new MCP tool; tool count stays 102. Co-Authored-By: Claude Fable 5 --- src/cli/brain/verbs/batch-plan.ts | 27 +++- src/cli/command-manifest.ts | 1 + src/core/brain/ingest/reconcile.ts | 80 ++++++++++++ src/mcp/brain/ingest-tools.ts | 26 +++- tests/cli/brain-batch-plan-reconcile.test.ts | 61 +++++++++ tests/core/brain/ingest/reconcile.test.ts | 124 +++++++++++++++++++ tests/mcp/ingest-tool.test.ts | 37 ++++++ 7 files changed, 353 insertions(+), 3 deletions(-) create mode 100644 src/core/brain/ingest/reconcile.ts create mode 100644 tests/cli/brain-batch-plan-reconcile.test.ts create mode 100644 tests/core/brain/ingest/reconcile.test.ts diff --git a/src/cli/brain/verbs/batch-plan.ts b/src/cli/brain/verbs/batch-plan.ts index 4a771914..ccb396d7 100644 --- a/src/cli/brain/verbs/batch-plan.ts +++ b/src/cli/brain/verbs/batch-plan.ts @@ -11,6 +11,7 @@ */ import { planBatches } from "../../../core/brain/ingest/batch-plan.ts"; +import { reconcilePlan } from "../../../core/brain/ingest/reconcile.ts"; import { brainVerbContext, fail, info, ok, okJson, parse, usageError } from "../helpers.ts"; /** Default caps, mirroring the MCP surface (1 MiB / 25 files per batch). */ @@ -34,13 +35,14 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise { "src-subpath": { type: "string" }, exclude: { type: "string-array" }, resume: { type: "boolean" }, + reconcile: { type: "boolean" }, json: { type: "boolean" }, }); const sourceDir = positional[0]; if (!sourceDir) { return usageError( "usage: o2b brain batch-plan [--max-bytes N] [--max-files N] " + - "[--src-subpath PATH] [--exclude PATTERN]... [--resume] [--json]", + "[--src-subpath PATH] [--exclude PATTERN]... [--resume] [--reconcile] [--json]", ); } @@ -51,6 +53,7 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise { const resume = flags["resume"] === true; const srcSubpath = flags["src-subpath"] as string | undefined; const exclude = (flags["exclude"] as string[] | undefined) ?? []; + const reconcile = flags["reconcile"] === true; const plan = planBatches(vault, sourceDir, { maxBatchBytes, maxBatchFiles, @@ -58,6 +61,9 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise { ...(srcSubpath !== undefined ? { srcSubpath } : {}), ...(exclude.length > 0 ? { exclude } : {}), }); + // The reconcile is read-only: it diffs the plan's dispatched set against the + // checkpoint and reports the gap; it never re-dispatches. + const report = reconcile ? reconcilePlan(vault, plan) : undefined; if (flags["json"]) { okJson({ @@ -84,6 +90,17 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise { total_bytes: b.totalBytes, files: b.files.map((f) => ({ path: f.path, bytes: f.bytes, status: f.status })), })), + // Only present with --reconcile, so the default output is unchanged. + ...(report !== undefined + ? { + reconcile: { + dispatched: [...report.dispatched], + ingested: [...report.ingested], + missing: [...report.missing], + complete: report.complete, + }, + } + : {}), }); return 0; } @@ -106,6 +123,14 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise { info(` ${plan.skippedNonExtractable.length} non-extractable page(s) skipped:`); for (const s of plan.skippedNonExtractable) info(` - ${s.path} (${s.reason})`); } + if (report !== undefined) { + if (report.complete) { + ok(` reconcile: all ${report.dispatched.length} dispatched source(s) ingested`); + } else { + info(` reconcile: ${report.missing.length} dispatched source(s) never ingested:`); + for (const p of report.missing) info(` - ${p}`); + } + } return 0; } catch (err) { return fail((err as Error).message ?? String(err)); diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.ts index 445359c1..2d4efe52 100644 --- a/src/cli/command-manifest.ts +++ b/src/cli/command-manifest.ts @@ -216,6 +216,7 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({ flag("vault", "string"), flag("max-bytes", "string"), flag("max-files", "string"), + flag("reconcile", "boolean"), flag("json", "boolean"), ], ), diff --git a/src/core/brain/ingest/reconcile.ts b/src/core/brain/ingest/reconcile.ts new file mode 100644 index 00000000..9aed3736 --- /dev/null +++ b/src/core/brain/ingest/reconcile.ts @@ -0,0 +1,80 @@ +/** + * Dispatched-vs-ingested reconciliation for batch plans (P5, t_d067a153). + * + * A large-folder ingest is planned by {@link planBatches} and dispatched as + * parallel subagents; each completed item folds into the plan's checkpoint via + * {@link ../ingest/ingest.ts:ingestSource}'s `planId`. Nothing, however, told + * the operator when the agent response quietly dropped a dispatched source. This + * module closes that loop: {@link reconcilePlan} diffs the plan's dispatched set + * against the checkpoint's completed entries and returns the gap. + * + * Report, not retry (a deliberate design decision): reconcile re-dispatches + * nothing. It only reads the checkpoint and returns/warns what is missing; + * re-ingesting the gap stays operator-driven. It is therefore read-only over + * checkpoint state and idempotent - the same plan and checkpoint always yield + * the same report, and calling it never mutates a byte on disk. + * + * The dispatched set is the full ingestible set the plan enumerated: the files + * still packed into batches PLUS the files the content-hash manifest classified + * `unchanged` (already ingested in a prior run of the same plan). Reconciling + * that union against the checkpoint means a plan re-run after a partial ingest + * still names exactly the sources that never landed. + */ + +import type { BatchPlan } from "./batch-plan.ts"; +import { readCheckpoint } from "./checkpoint.ts"; + +/** The gap report for one plan: what was dispatched vs what actually ingested. */ +export interface ReconcileReport { + /** The plan id the report is keyed on (matches {@link BatchPlan.planId}). */ + readonly planId: string; + /** Every ingestible source the plan enumerated, sorted and deduped. */ + readonly dispatched: readonly string[]; + /** Dispatched sources the checkpoint confirms ingested, sorted. */ + readonly ingested: readonly string[]; + /** Dispatched sources the checkpoint never recorded - the gap, sorted. */ + readonly missing: readonly string[]; + /** True when the gap is empty (every dispatched source ingested). */ + readonly complete: boolean; +} + +/** + * Reconcile a batch plan against its checkpoint. Returns the set of dispatched + * sources that never reached the checkpoint's completed set. Read-only and + * idempotent: it reads the checkpoint keyed on `plan.planId` and mutates + * nothing. A plan that dispatched nothing, or whose sources all completed, + * reports an empty (explicitly `complete`) gap. + */ +export function reconcilePlan(vault: string, plan: BatchPlan): ReconcileReport { + const dispatched = collectDispatched(plan); + const completed = new Set(readCheckpoint(vault, plan.planId)?.completed ?? []); + + const ingested: string[] = []; + const missing: string[] = []; + for (const path of dispatched) { + if (completed.has(path)) ingested.push(path); + else missing.push(path); + } + + return { + planId: plan.planId, + dispatched, + ingested, + missing, + complete: missing.length === 0, + }; +} + +/** + * The dispatched set: the batch files plus the manifest-`unchanged` skips, as a + * sorted, deduped list. `skippedNonExtractable` is excluded - those pages were + * deliberately gated out before dispatch, so they were never part of the work. + */ +function collectDispatched(plan: BatchPlan): string[] { + const set = new Set(); + for (const batch of plan.batches) { + for (const file of batch.files) set.add(file.path); + } + for (const path of plan.skipped) set.add(path); + return [...set].toSorted(); +} diff --git a/src/mcp/brain/ingest-tools.ts b/src/mcp/brain/ingest-tools.ts index f5311660..401c8594 100644 --- a/src/mcp/brain/ingest-tools.ts +++ b/src/mcp/brain/ingest-tools.ts @@ -12,6 +12,7 @@ import { planBatches, type BatchPlan } from "../../core/brain/ingest/batch-plan.ts"; import { clearCheckpoint } from "../../core/brain/ingest/checkpoint.ts"; import { ingestSource } from "../../core/brain/ingest/ingest.ts"; +import { reconcilePlan } from "../../core/brain/ingest/reconcile.ts"; import { IntakeValidationError } from "../../core/brain/intake/extract-intake.ts"; import { deleteBySource, @@ -210,6 +211,7 @@ async function toolBrainIngestBatchPlan( Number.MAX_SAFE_INTEGER, ); const resume = coerceBoolOptional(args, "resume") ?? false; + const reconcile = coerceBoolOptional(args, "reconcile") ?? false; const srcSubpath = coerceStr(args, "src_subpath", false) ?? undefined; const exclude = coerceStrList(args, "exclude"); const plan = planBatches(ctx.vault, sourceDir, { @@ -219,12 +221,27 @@ async function toolBrainIngestBatchPlan( ...(srcSubpath !== undefined ? { srcSubpath } : {}), ...(exclude.length > 0 ? { exclude } : {}), }); + // Reconcile BEFORE any checkpoint clear below, so the gap report reads a live + // checkpoint. Read-only: it only diffs dispatched vs completed. + const report = reconcile ? reconcilePlan(ctx.vault, plan) : undefined; // A resumed plan that comes back empty is fully drained: drop its checkpoint // (the content manifest is the authoritative final state from here on). if (resume && plan.batches.length === 0) { clearCheckpoint(ctx.vault, plan.planId); } - return serializeBatchPlan(plan); + const serialized = serializeBatchPlan(plan); + return report === undefined + ? serialized + : { + ...serialized, + reconcile: { + plan_id: report.planId, + dispatched: [...report.dispatched], + ingested: [...report.ingested], + missing: [...report.missing], + complete: report.complete, + }, + }; } export const INGEST_TOOLS: ReadonlyArray = Object.freeze([ @@ -286,7 +303,7 @@ export const INGEST_TOOLS: ReadonlyArray = Object.freeze([ pre_extract: { type: "boolean", description: - "Run the deterministic no-LLM code-structure pass over the source file and return its class/function/import/inheritance seeds under `pre_extract`. Off by default; absent leaves the response unchanged.", + "Run the deterministic no-LLM code-structure pass over the source file; returns class/function/import/inheritance seeds under `pre_extract`. Off by default.", }, }, required: ["source_path", "summary", "entities"], @@ -378,6 +395,11 @@ export const INGEST_TOOLS: ReadonlyArray = Object.freeze([ description: "Resume an interrupted plan: exclude items already recorded completed in this plan's checkpoint. Default false.", }, + reconcile: { + type: "boolean", + description: + "Also return a `reconcile` gap report diffing the plan's dispatched sources against the checkpoint's ingested set. Read-only; no retry. Default false.", + }, src_subpath: { type: "string", description: diff --git a/tests/cli/brain-batch-plan-reconcile.test.ts b/tests/cli/brain-batch-plan-reconcile.test.ts new file mode 100644 index 00000000..b60a906d --- /dev/null +++ b/tests/cli/brain-batch-plan-reconcile.test.ts @@ -0,0 +1,61 @@ +/** + * P5 (t_d067a153): `o2b brain batch-plan --reconcile` plumbs the dispatched-vs- + * ingested gap report through to the CLI. The reconcile logic is covered at the + * core level; this asserts the flag reaches the reconciler, the report rides on + * the JSON output, and omitting the flag leaves the output unchanged. + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { runCli } from "../helpers/run-cli.ts"; + +let vault: string; +let configHome: string; +let configPath: string; + +beforeEach(() => { + vault = mkdtempSync(join(tmpdir(), "o2b-reconcile-cli-vault-")); + configHome = mkdtempSync(join(tmpdir(), "o2b-reconcile-cli-cfg-")); + configPath = join(configHome, "config.yaml"); + writeFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`, "utf8"); +}); + +afterEach(() => { + rmSync(vault, { recursive: true, force: true }); + rmSync(configHome, { recursive: true, force: true }); +}); + +function write(rel: string): void { + const abs = join(vault, rel); + mkdirSync(join(abs, ".."), { recursive: true }); + writeFileSync(abs, "content\n", "utf8"); +} + +const ENV = () => ({ OPEN_SECOND_BRAIN_CONFIG: configPath }); + +describe("o2b brain batch-plan --reconcile", () => { + test("emits a gap report naming un-ingested dispatched sources", async () => { + write("Docs/a.md"); + write("Docs/b.md"); + // No ingest has happened, so every dispatched source is in the gap. + const res = await runCli(["brain", "batch-plan", "Docs", "--reconcile", "--json"], { + env: ENV(), + }); + expect(res.returncode).toBe(0); + const plan = JSON.parse(res.stdout); + expect(plan.reconcile.dispatched).toEqual(["Docs/a.md", "Docs/b.md"]); + expect(plan.reconcile.missing).toEqual(["Docs/a.md", "Docs/b.md"]); + expect(plan.reconcile.complete).toBe(false); + }); + + test("without the flag the JSON omits the reconcile report", async () => { + write("Docs/a.md"); + const res = await runCli(["brain", "batch-plan", "Docs", "--json"], { env: ENV() }); + expect(res.returncode).toBe(0); + const plan = JSON.parse(res.stdout); + expect(plan.reconcile).toBeUndefined(); + }); +}); diff --git a/tests/core/brain/ingest/reconcile.test.ts b/tests/core/brain/ingest/reconcile.test.ts new file mode 100644 index 00000000..c5bbe072 --- /dev/null +++ b/tests/core/brain/ingest/reconcile.test.ts @@ -0,0 +1,124 @@ +/** + * P5 (t_d067a153): dispatched-vs-ingested reconciliation for batch plans. + * + * After a batch plan drains, `reconcilePlan` diffs the plan's dispatched set + * against the checkpoint's completed entries and reports the gap - each source + * that was dispatched but never recorded as ingested. It is a WARNING surface, + * not a retry: it re-dispatches nothing, is read-only over checkpoint state, and + * is idempotent. A fully completed plan reports an empty gap explicitly. + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { existsSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { bootstrapBrain } from "../../../../src/core/brain/init.ts"; +import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts"; +import { ingestSource } from "../../../../src/core/brain/ingest/ingest.ts"; +import { planBatches } from "../../../../src/core/brain/ingest/batch-plan.ts"; +import { checkpointPath } from "../../../../src/core/brain/ingest/checkpoint.ts"; +import { reconcilePlan } from "../../../../src/core/brain/ingest/reconcile.ts"; + +let vault: string; +let configHome: string; + +const NOW = new Date("2026-06-13T12:00:00Z"); +const CAPS = { maxBatchBytes: 100_000, maxBatchFiles: 100 } as const; + +beforeEach(() => { + vault = mkdtempSync(join(tmpdir(), "o2b-reconcile-vault-")); + configHome = mkdtempSync(join(tmpdir(), "o2b-reconcile-cfg-")); + const configPath = join(configHome, "config.yaml"); + atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`); + bootstrapBrain(vault, { configPath }); +}); + +afterEach(() => { + rmSync(vault, { recursive: true, force: true }); + rmSync(configHome, { recursive: true, force: true }); +}); + +function write(rel: string, content = "content\n"): void { + const abs = join(vault, rel); + mkdirSync(join(abs, ".."), { recursive: true }); + writeFileSync(abs, content, "utf8"); +} + +function ingest(rel: string, planId: string): void { + ingestSource( + vault, + { + sourcePath: rel, + summary: `Summary of ${rel}.`, + extraction: { entities: [{ category: "concept", name: rel }], relations: [] }, + }, + { agent: "claude", now: NOW, planId }, + ); +} + +describe("reconcilePlan", () => { + test("names each dispatched source the checkpoint never recorded", () => { + write("Docs/a.md"); + write("Docs/b.md"); + const plan = planBatches(vault, "Docs", CAPS); + ingest("Docs/a.md", plan.planId); // only a completes; b is lost + + const report = reconcilePlan(vault, plan); + expect(report.planId).toBe(plan.planId); + expect(report.dispatched).toEqual(["Docs/a.md", "Docs/b.md"]); + expect(report.ingested).toEqual(["Docs/a.md"]); + expect(report.missing).toEqual(["Docs/b.md"]); + expect(report.complete).toBe(false); + }); + + test("a fully completed plan reports an empty gap explicitly", () => { + write("Docs/a.md"); + write("Docs/b.md"); + const plan = planBatches(vault, "Docs", CAPS); + ingest("Docs/a.md", plan.planId); + ingest("Docs/b.md", plan.planId); + + const report = reconcilePlan(vault, plan); + expect(report.missing).toEqual([]); + expect(report.complete).toBe(true); + }); + + test("a plan with no dispatched files reports a clean empty gap", () => { + // No ingestible files under the dir -> nothing dispatched, nothing missing. + write("Docs/notes.bin"); + const plan = planBatches(vault, "Docs", CAPS); + const report = reconcilePlan(vault, plan); + expect(report.dispatched).toEqual([]); + expect(report.missing).toEqual([]); + expect(report.complete).toBe(true); + }); + + test("with no checkpoint every dispatched source is missing", () => { + write("Docs/a.md"); + write("Docs/b.md"); + const plan = planBatches(vault, "Docs", CAPS); + const report = reconcilePlan(vault, plan); + expect(report.missing).toEqual(["Docs/a.md", "Docs/b.md"]); + expect(report.ingested).toEqual([]); + expect(report.complete).toBe(false); + }); + + test("is idempotent and read-only over checkpoint state", () => { + write("Docs/a.md"); + write("Docs/b.md"); + const plan = planBatches(vault, "Docs", CAPS); + ingest("Docs/a.md", plan.planId); + + const cpPath = checkpointPath(vault, plan.planId); + expect(existsSync(cpPath)).toBe(true); + const before = readFileSync(cpPath, "utf8"); + + const first = reconcilePlan(vault, plan); + const second = reconcilePlan(vault, plan); + expect(JSON.stringify(first)).toBe(JSON.stringify(second)); + // The reconcile must not write, clear, or otherwise touch the checkpoint. + expect(readFileSync(cpPath, "utf8")).toBe(before); + }); +}); diff --git a/tests/mcp/ingest-tool.test.ts b/tests/mcp/ingest-tool.test.ts index 6a293031..f5347967 100644 --- a/tests/mcp/ingest-tool.test.ts +++ b/tests/mcp/ingest-tool.test.ts @@ -143,3 +143,40 @@ describe("brain_ingest_batch_plan resume (t_ba1fa5f6)", () => { expect(files).toEqual(["Docs/b.md"]); }); }); + +describe("brain_ingest_batch_plan reconcile (P5, t_d067a153)", () => { + const batchPlan = INGEST_TOOLS.find((t) => t.name === "brain_ingest_batch_plan")!.handler; + + test("reconcile reports sources dispatched but never ingested", async () => { + mkdirSync(join(vault, "Docs"), { recursive: true }); + writeFileSync(join(vault, "Docs", "a.md"), "alpha", "utf8"); + writeFileSync(join(vault, "Docs", "b.md"), "bravo", "utf8"); + + const first = (await batchPlan(ctx, { source_dir: "Docs" })) as Record; + // Ingest only a; b is the lost source. + await handler(ctx, { + source_path: "Docs/a.md", + summary: "Alpha.", + entities: [{ category: "concept", name: "Alpha" }], + plan_id: first["plan_id"], + }); + + const res = (await batchPlan(ctx, { source_dir: "Docs", reconcile: true })) as Record< + string, + unknown + >; + expect(res["reconcile"]).toMatchObject({ + plan_id: first["plan_id"], + ingested: ["Docs/a.md"], + missing: ["Docs/b.md"], + complete: false, + }); + }); + + test("without the reconcile flag the response omits the report (byte-identical)", async () => { + mkdirSync(join(vault, "Docs"), { recursive: true }); + writeFileSync(join(vault, "Docs", "a.md"), "alpha", "utf8"); + const res = (await batchPlan(ctx, { source_dir: "Docs" })) as Record; + expect(res["reconcile"]).toBeUndefined(); + }); +}); From f1be8c10862ad1c2fe5c677bef572711a09c1f0e Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 21:32:12 +0000 Subject: [PATCH 08/14] feat(temporal): promote inline source citations to dated provenance events Add a structural parser for inline `[Source: , YYYY-MM-DD]` prose markers and a scan surface that promotes each well-formed citation into the temporal timeline as a dated `source-citation` provenance event, stamped at the citation date. - New `src/core/brain/temporal/citations.ts`: pure `parseCitations` (fixed marker grammar, ISO-shaped date validated as a real calendar date, no natural-language date parsing) plus `scanCitations` which walks the configured note folders and appends one dated event per unique citation. - Dedup on (normalized name, date) against already-logged source-citation events, so a re-scan of an unchanged vault promotes nothing. Malformed markers are reported explicitly and skipped. - New `source-citation` log event kind; a new `o2b brain scan-citations` CLI verb (--dry-run, --strict, --path, --exclude, --json). Kept CLI-only for the agent surface: scanning is an operator maintenance action like scan-inline (also CLI-only), and adding an MCP tool would break the frozen 102-tool parity count with no existing tool to extend. Co-Authored-By: Claude Fable 5 --- src/cli/brain.ts | 3 + src/cli/brain/help-text.ts | 11 + src/cli/brain/verbs/index.ts | 1 + src/cli/brain/verbs/scan-citations.ts | 61 ++++ src/cli/command-manifest.ts | 1 + src/core/brain/temporal/citations.ts | 367 ++++++++++++++++++++ src/core/brain/types.ts | 12 + tests/core/brain.types.test.ts | 2 + tests/core/brain/temporal/citations.test.ts | 179 ++++++++++ 9 files changed, 637 insertions(+) create mode 100644 src/cli/brain/verbs/scan-citations.ts create mode 100644 src/core/brain/temporal/citations.ts create mode 100644 tests/core/brain/temporal/citations.test.ts diff --git a/src/cli/brain.ts b/src/cli/brain.ts index 85423179..8c57b7b4 100644 --- a/src/cli/brain.ts +++ b/src/cli/brain.ts @@ -77,6 +77,7 @@ import { cmdBrainUpgrade, handleBrainSnapshotSubcommand, cmdBrainScanInline, + cmdBrainScanCitations, cmdBrainEntity, cmdBrainImportSession, cmdBrainForgetSource, @@ -286,6 +287,8 @@ export async function handleBrainSubcommand(argv: ReadonlyArray): Promis return await cmdBrainMcpLandscape(rest); case "scan-inline": return await cmdBrainScanInline(rest); + case "scan-citations": + return cmdBrainScanCitations(rest); case "import-session": return await cmdBrainImportSession(rest); case "handoff": diff --git a/src/cli/brain/help-text.ts b/src/cli/brain/help-text.ts index 63083b2c..d4a76ad4 100644 --- a/src/cli/brain/help-text.ts +++ b/src/cli/brain/help-text.ts @@ -79,6 +79,7 @@ Brain verbs (observing memory): authored-at-backfill Backfill authored_at on session signals (dry-run default; --apply to write) mcp-landscape List MCP servers configured across the vault (packages, env names) scan-inline Capture @osb markers from folders listed under notes.read_paths in _brain.yaml + scan-citations Promote inline [Source: , YYYY-MM-DD] markers to dated provenance events import-session Replay signals from a registered agent session .jsonl (or directory) entity Canonical entity registry: set, get, list, relate, archive, prune session-hook Capture one runtime hook payload from stdin (internal hook bridge) @@ -536,6 +537,16 @@ export const VERB_HELP: Record = { "create signals in Brain/inbox/, and annotate the source files with\n" + "@osb✓ [[sig-...]]. Brain/, .git, node_modules, and similar directories\n" + "are always skipped. Idempotent on re-run.\n", + "scan-citations": + "usage: o2b brain scan-citations [--vault ] [--path ...] [--exclude ...]\n" + + " [--dry-run] [--strict] [--json] [--agent ]\n" + + "Walk the vault for inline [Source: , YYYY-MM-DD] provenance markers\n" + + "and promote each well-formed citation into the temporal timeline as a\n" + + "dated source-citation event stamped at the citation date. Dedup is on\n" + + "(normalized name, date), so a re-run over an unchanged vault promotes\n" + + "nothing. Malformed markers are reported and skipped; --strict exits 2\n" + + "when any malformed marker is found. Brain/ and the usual build dirs are\n" + + "always skipped.\n", "import-claude-memory": "usage: o2b brain import-claude-memory [--vault ] [--memory ]\n" + " [--dry-run | --apply] [--yes] [--json]\n" + diff --git a/src/cli/brain/verbs/index.ts b/src/cli/brain/verbs/index.ts index 973d8085..1c97531e 100644 --- a/src/cli/brain/verbs/index.ts +++ b/src/cli/brain/verbs/index.ts @@ -61,6 +61,7 @@ export { cmdBrainOkfImport } from "./okf-import.ts"; export { cmdBrainUpgrade } from "./upgrade.ts"; export { cmdBrainSnapshotDiff, handleBrainSnapshotSubcommand } from "./snapshot.ts"; export { cmdBrainScanInline } from "./scan-inline.ts"; +export { cmdBrainScanCitations } from "./scan-citations.ts"; export { cmdBrainImportSession } from "./import-session.ts"; export { cmdBrainHandoff } from "./handoff.ts"; export { cmdBrainIntention } from "./intention.ts"; diff --git a/src/cli/brain/verbs/scan-citations.ts b/src/cli/brain/verbs/scan-citations.ts new file mode 100644 index 00000000..38560b49 --- /dev/null +++ b/src/cli/brain/verbs/scan-citations.ts @@ -0,0 +1,61 @@ +import { scanCitations } from "../../../core/brain/temporal/citations.ts"; +import { brainVerbContext, fail, info, ok, okJson, parse, resolveBrainAgent } from "../helpers.ts"; + +export function cmdBrainScanCitations(argv: string[]): number { + const { flags } = parse(argv, { + vault: { type: "string" }, + "dry-run": { type: "boolean" }, + strict: { type: "boolean" }, + path: { type: "string-array" }, + exclude: { type: "string-array" }, + agent: { type: "string" }, + json: { type: "boolean" }, + }); + const { config, vault } = brainVerbContext(flags); + const agent = resolveBrainAgent(flags, config); + + let result; + try { + result = scanCitations(vault, { + agent, + dryRun: Boolean(flags["dry-run"]), + paths: (flags["path"] as string[] | undefined) ?? [], + exclude: (flags["exclude"] as string[] | undefined) ?? [], + }); + } catch (exc) { + return fail(`scan-citations failed: ${(exc as Error).message ?? exc}`); + } + + if (flags["json"]) { + okJson({ + scanned: result.scanned, + found: result.found, + promoted: result.promoted, + deduped: result.deduped, + malformed: result.malformed, + malformed_markers: result.malformedMarkers.map((m) => ({ + path: m.path, + line: m.line, + raw: m.raw, + reason: m.reason, + })), + errors: result.errors.map((e) => ({ path: e.path, message: e.message })), + files_with_citations: result.filesWithCitations.map((f) => ({ + path: f.path, + citations: f.citations, + })), + }); + } else { + ok(`scanned: ${result.scanned}`); + ok(`found: ${result.found}`); + ok(`promoted: ${result.promoted}`); + ok(`deduped: ${result.deduped}`); + if (result.malformed > 0) ok(`malformed: ${result.malformed}`); + for (const m of result.malformedMarkers) { + info(` malformed: ${m.path}:${m.line}: ${m.reason} (${m.raw})`); + } + for (const e of result.errors) info(` error: ${e.path}: ${e.message}`); + } + if (flags["strict"] && result.malformed > 0) return 2; + return 0; +} diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.ts index 2d4efe52..4806eb9c 100644 --- a/src/cli/command-manifest.ts +++ b/src/cli/command-manifest.ts @@ -194,6 +194,7 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({ command("semantics-backfill", "Preview Brain semantics backfill"), command("mcp-landscape", "List MCP servers configured across the vault"), command("scan-inline", "Capture inline @osb markers"), + command("scan-citations", "Promote inline [Source: ...] citations to dated events"), command("import-session", "Replay registered agent sessions"), command("handoff", "Write an operator-readable session handoff note"), command("intention", "Manage scoped current-intention chains"), diff --git a/src/core/brain/temporal/citations.ts b/src/core/brain/temporal/citations.ts new file mode 100644 index 00000000..02eaa78c --- /dev/null +++ b/src/core/brain/temporal/citations.ts @@ -0,0 +1,367 @@ +/** + * Inline source-citation promotion (Source pipeline integrity suite, + * Q1, t_a3d1adb0). + * + * Notes carry prose provenance as a fixed structural marker: + * + * [Source: , YYYY-MM-DD] + * + * This module parses that marker out of note prose and promotes each + * well-formed citation into the temporal timeline as a dated + * `source-citation` provenance event, stamped at the citation date so it + * lands on the timeline where the source was dated rather than when it + * was scanned. + * + * The marker is a FIXED grammar (the literal `[Source:` prefix and a + * comma separator), never a natural-language heuristic. The date is + * parsed purely structurally: an ISO `YYYY-MM-DD` shape validated as a + * real calendar date. Month names or other localized date forms are NOT + * recognised - they surface as malformed markers, never silently parsed. + * + * Dedup key is (normalized name, date) against already-logged + * `source-citation` events, so a re-scan of an unchanged vault promotes + * nothing and leaves the log byte-identical. Malformed markers are + * reported explicitly and skipped; they never abort the surrounding scan. + */ + +import { readFileSync } from "node:fs"; +import { relative, sep } from "node:path"; + +import { appendLogEvent } from "../log.ts"; +import { readLogDay } from "../log-jsonl.ts"; +import { buildNoteWalkRules, resolveNoteRoots, walkMarkdownFiles } from "../notes/note-walk.ts"; +import { BRAIN_LOG_EVENT_KIND } from "../types.ts"; + +/** Per-file scan cap, matching the inline-marker scanner (1 MiB). */ +const MAX_FILE_SIZE_BYTES = 1_048_576; + +/** + * Candidate marker: the literal `[Source:` prefix, an inner payload with + * no closing bracket, then `]`. The inner payload is validated + * separately so a bracket-shaped-but-invalid marker is reported as + * malformed rather than missed. + */ +const CANDIDATE_RE = /\[Source:\s*([^\]]*)\]/g; + +/** + * Structural split of a candidate's inner payload into `` and an + * ISO `YYYY-MM-DD` date anchored at the end. The name is everything up + * to the LAST comma that precedes the trailing date, so names may + * themselves contain commas. The date shape is validated for real + * calendar bounds by {@link isRealIsoDate}. + */ +const INNER_RE = /^(.*\S)\s*,\s*(\d{4})-(\d{2})-(\d{2})$/; + +const DAYS_IN_MONTH: ReadonlyArray = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]; + +/** One well-formed citation marker parsed from note prose. */ +export interface CitationMarker { + /** Raw citation name, trimmed. Whitespace preserved as written. */ + readonly name: string; + /** ISO `YYYY-MM-DD` citation date (validated as a real calendar date). */ + readonly date: string; + /** 1-based line number the marker was found on. */ + readonly line: number; +} + +/** A bracket-shaped marker that failed structural validation. */ +export interface MalformedCitation { + /** 1-based line number the candidate was found on. */ + readonly line: number; + /** The raw candidate text, verbatim. */ + readonly raw: string; + /** Structural reason the candidate was rejected. */ + readonly reason: string; +} + +export interface CitationParseResult { + readonly markers: ReadonlyArray; + readonly malformed: ReadonlyArray; +} + +/** + * Normalize a citation name for dedup: trim, collapse internal runs of + * whitespace to a single space, and case-fold. Case folding is Unicode + * general (`toLowerCase`), never a per-language table. + */ +export function normalizeCitationName(name: string): string { + return name.trim().replace(/\s+/g, " ").toLowerCase(); +} + +/** True when `y-m-d` is a real calendar date (leap years accounted for). */ +function isRealIsoDate(y: number, m: number, d: number): boolean { + if (m < 1 || m > 12) return false; + if (d < 1) return false; + const leap = (y % 4 === 0 && y % 100 !== 0) || y % 400 === 0; + const max = m === 2 && leap ? 29 : DAYS_IN_MONTH[m - 1]!; + return d <= max; +} + +/** + * Parse every `[Source: , YYYY-MM-DD]` marker out of `content`. + * Well-formed markers land in `markers`; bracket-shaped candidates that + * fail structural validation land in `malformed` with a reason. Pure and + * deterministic; performs no I/O. + */ +export function parseCitations(content: string): CitationParseResult { + const markers: CitationMarker[] = []; + const malformed: MalformedCitation[] = []; + const lines = content.split(/\r?\n/); + + for (let i = 0; i < lines.length; i++) { + const line = lines[i]!; + const lineNumber = i + 1; + CANDIDATE_RE.lastIndex = 0; + let match: RegExpExecArray | null; + while ((match = CANDIDATE_RE.exec(line)) !== null) { + const raw = match[0]!; + const inner = match[1]!.trim(); + const parsed = INNER_RE.exec(inner); + if (parsed === null) { + malformed.push({ + line: lineNumber, + raw, + reason: "expected ', YYYY-MM-DD' (missing comma or non-ISO date shape)", + }); + continue; + } + const name = parsed[1]!.trim(); + const year = Number(parsed[2]!); + const month = Number(parsed[3]!); + const day = Number(parsed[4]!); + if (name.length === 0) { + malformed.push({ line: lineNumber, raw, reason: "empty citation name" }); + continue; + } + if (!isRealIsoDate(year, month, day)) { + malformed.push({ + line: lineNumber, + raw, + reason: `not a real calendar date: ${parsed[2]}-${parsed[3]}-${parsed[4]}`, + }); + continue; + } + markers.push({ name, date: `${parsed[2]}-${parsed[3]}-${parsed[4]}`, line: lineNumber }); + } + } + + return Object.freeze({ + markers: Object.freeze(markers), + malformed: Object.freeze(malformed), + }); +} + +export interface ScanCitationsOptions { + /** Agent identity stamped on every promoted event. */ + readonly agent: string; + /** When true, report promotions without writing any events. */ + readonly dryRun?: boolean; + /** Narrow the walker to vault-relative subdirs only. */ + readonly paths?: ReadonlyArray; + /** Additional vault-relative exclude prefixes. */ + readonly exclude?: ReadonlyArray; +} + +export interface CitationScanError { + readonly path: string; + readonly message: string; +} + +export interface CitationScanMalformed { + readonly path: string; + readonly line: number; + readonly raw: string; + readonly reason: string; +} + +export interface CitationScanFileSummary { + readonly path: string; + readonly citations: number; +} + +export interface CitationScanResult { + /** Files walked. */ + readonly scanned: number; + /** Well-formed markers found. */ + readonly found: number; + /** Events written (0 on a dry run). */ + readonly promoted: number; + /** Well-formed markers skipped as duplicates of already-logged events. */ + readonly deduped: number; + /** Malformed marker count. */ + readonly malformed: number; + readonly malformedMarkers: ReadonlyArray; + readonly errors: ReadonlyArray; + readonly filesWithCitations: ReadonlyArray; +} + +/** + * Lazily-built per-date set of `${normalizedName}` keys already present + * as `source-citation` events in that day's log. Because a promoted + * event is always stamped at its citation date, dedup only needs that + * one day's log rather than the whole timeline. + */ +class DedupIndex { + private readonly byDate = new Map>(); + + constructor(private readonly vault: string) {} + + private forDate(date: string): Set { + const cached = this.byDate.get(date); + if (cached !== undefined) return cached; + const seen = new Set(); + const { entries } = readLogDay(this.vault, date); + for (const entry of entries) { + if (entry.eventType !== BRAIN_LOG_EVENT_KIND.sourceCitation) continue; + const name = entry.body["name"]; + const at = entry.body["date"]; + if (typeof name === "string" && typeof at === "string" && at === date) { + seen.add(normalizeCitationName(name)); + } + } + this.byDate.set(date, seen); + return seen; + } + + has(name: string, date: string): boolean { + return this.forDate(date).has(normalizeCitationName(name)); + } + + add(name: string, date: string): void { + this.forDate(date).add(normalizeCitationName(name)); + } +} + +/** + * Walk the configured note folders, parse citation markers, and promote + * each unique well-formed citation into a dated `source-citation` event. + * Read-only when `dryRun` is set. Malformed markers and per-file read + * errors are collected and reported, never silently swallowed. + */ +export function scanCitations(vault: string, opts: ScanCitationsOptions): CitationScanResult { + const errors: CitationScanError[] = []; + const malformedMarkers: CitationScanMalformed[] = []; + const filesWithCitations: CitationScanFileSummary[] = []; + + let scanned = 0; + let found = 0; + let promoted = 0; + let deduped = 0; + + const roots = resolveNoteRoots(vault, opts.paths); + if (roots.length === 0) { + return freezeResult({ + scanned, + found, + promoted, + deduped, + malformedMarkers, + errors, + filesWithCitations, + }); + } + + const rules = buildNoteWalkRules(vault, opts.exclude); + const dedup = new DedupIndex(vault); + + for (const file of walkMarkdownFiles(vault, roots, rules, { + maxFileSizeBytes: MAX_FILE_SIZE_BYTES, + onOversize: (oversize, size) => { + errors.push({ + path: oversize.absPath, + message: `file too large to scan (${size} bytes; cap ${MAX_FILE_SIZE_BYTES})`, + }); + }, + })) { + scanned++; + let content: string; + try { + content = readFileSync(file.absPath, "utf8"); + } catch (err) { + errors.push({ + path: file.absPath, + message: `read failed: ${(err as Error).message ?? String(err)}`, + }); + continue; + } + + const parsed = parseCitations(content); + for (const bad of parsed.malformed) { + malformedMarkers.push({ + path: file.absPath, + line: bad.line, + raw: bad.raw, + reason: bad.reason, + }); + } + if (parsed.markers.length === 0) continue; + found += parsed.markers.length; + filesWithCitations.push({ path: file.absPath, citations: parsed.markers.length }); + + const vaultRelSource = relative(vault, file.absPath).split(sep).join("/"); + for (const marker of parsed.markers) { + if (dedup.has(marker.name, marker.date)) { + deduped++; + continue; + } + if (opts.dryRun) { + // Count the promotion the run WOULD make, and record it in the + // in-memory index so repeated markers within one dry run still + // dedup against the first. + promoted++; + dedup.add(marker.name, marker.date); + continue; + } + try { + appendLogEvent(vault, { + timestamp: `${marker.date}T00:00:00Z`, + eventType: BRAIN_LOG_EVENT_KIND.sourceCitation, + body: { + agent: opts.agent, + name: marker.name, + date: marker.date, + source: `[[${vaultRelSource}]]`, + }, + }); + dedup.add(marker.name, marker.date); + promoted++; + } catch (err) { + errors.push({ + path: file.absPath, + message: `promote failed: ${(err as Error).message ?? String(err)}`, + }); + } + } + } + + return freezeResult({ + scanned, + found, + promoted, + deduped, + malformedMarkers, + errors, + filesWithCitations, + }); +} + +function freezeResult(r: { + scanned: number; + found: number; + promoted: number; + deduped: number; + malformedMarkers: CitationScanMalformed[]; + errors: CitationScanError[]; + filesWithCitations: CitationScanFileSummary[]; +}): CitationScanResult { + return Object.freeze({ + scanned: r.scanned, + found: r.found, + promoted: r.promoted, + deduped: r.deduped, + malformed: r.malformedMarkers.length, + malformedMarkers: Object.freeze(r.malformedMarkers), + errors: Object.freeze(r.errors), + filesWithCitations: Object.freeze(r.filesWithCitations), + }); +} diff --git a/src/core/brain/types.ts b/src/core/brain/types.ts index ef2b58f7..5a6b53f9 100644 --- a/src/core/brain/types.ts +++ b/src/core/brain/types.ts @@ -380,6 +380,18 @@ export const BRAIN_LOG_EVENT_KIND = { * first detection and each deliberate transition. */ tension: "tension", + /** + * `source-citation` (Source pipeline integrity suite, Q1, t_a3d1adb0) - + * an inline `[Source: , YYYY-MM-DD]` prose citation was promoted + * into the temporal timeline as a dated provenance event. The event is + * stamped at the citation date (`T00:00:00Z`), so it lands on the + * timeline where the source was dated rather than when it was scanned. + * Payload carries the raw `name`, the ISO `date`, the vault-relative + * `source` note wikilink, and the `agent`. Dedup is on (normalized name, + * date) against already-logged source-citation events, so a re-scan of + * an unchanged vault promotes nothing and stays byte-identical. + */ + sourceCitation: "source-citation", } as const; export type BrainLogEventKind = (typeof BRAIN_LOG_EVENT_KIND)[keyof typeof BRAIN_LOG_EVENT_KIND]; diff --git a/tests/core/brain.types.test.ts b/tests/core/brain.types.test.ts index 4043ab97..ff5cb2c2 100644 --- a/tests/core/brain.types.test.ts +++ b/tests/core/brain.types.test.ts @@ -114,6 +114,8 @@ describe("BRAIN_* const enums", () => { "authored-at-backfill", // Belief lifecycle suite (t_0e3f2bee) tension object detect + transitions "tension", + // Source pipeline integrity suite (t_a3d1adb0) inline citation promotion + "source-citation", ]); const actual = new Set(Object.values(BRAIN_LOG_EVENT_KIND)); expect(actual).toEqual(expected); diff --git a/tests/core/brain/temporal/citations.test.ts b/tests/core/brain/temporal/citations.test.ts new file mode 100644 index 00000000..f6e2c673 --- /dev/null +++ b/tests/core/brain/temporal/citations.test.ts @@ -0,0 +1,179 @@ +/** + * Task Q1 (t_a3d1adb0): inline `[Source: , YYYY-MM-DD]` citation + * promotion into the temporal timeline as dated provenance events. + * + * Acceptance coverage: + * - a well-formed marker produces a dated provenance event on the + * timeline (event dated at the citation date, not scan time); + * - re-scanning does not duplicate (dedup on normalized name + date + * against already-logged source events); + * - malformed markers (bad date shape, missing comma) are reported + * explicitly and skipped; + * - notes without markers produce no events; + * - parsing is purely structural (no natural-language date parsing). + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { brainDirs } from "../../../../src/core/brain/paths.ts"; +import { DEFAULT_BRAIN_CONFIG_YAML } from "../../../../src/core/brain/policy.ts"; +import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts"; +import { buildTimelineIndex } from "../../../../src/core/brain/temporal/build-index.ts"; +import { BRAIN_LOG_EVENT_KIND } from "../../../../src/core/brain/types.ts"; +import { + normalizeCitationName, + parseCitations, + scanCitations, +} from "../../../../src/core/brain/temporal/citations.ts"; + +let tmp: string; + +beforeEach(() => { + tmp = mkdtempSync(join(tmpdir(), "o2b-citations-")); + const dirs = brainDirs(tmp); + for (const d of [dirs.brain, dirs.log]) mkdirSync(d, { recursive: true }); + atomicWriteFileSync( + join(dirs.brain, "_brain.yaml"), + `${DEFAULT_BRAIN_CONFIG_YAML}\nnotes:\n read_paths:\n - Notes\n`, + ); +}); + +afterEach(() => { + rmSync(tmp, { recursive: true, force: true }); +}); + +function writeMd(rel: string, content: string): void { + const path = join(tmp, rel); + mkdirSync(join(path, ".."), { recursive: true }); + writeFileSync(path, content, "utf8"); +} + +function sourceCitationEvents(vault: string) { + // Wide window: promoted events are stamped at the citation date, which + // for historical sources predates the timeline's default 1970 lower + // bound, so the verification window opens earlier. + const idx = buildTimelineIndex(vault, { since: "1000-01-01", until: "2999-01-01" }); + return idx.events.filter((e) => e.kind === BRAIN_LOG_EVENT_KIND.sourceCitation); +} + +describe("parseCitations (pure structural parser)", () => { + test("extracts a well-formed marker with name and ISO date", () => { + const res = parseCitations("Body text. [Source: Kant Critique, 1781-05-01] more.\n"); + expect(res.markers).toHaveLength(1); + expect(res.markers[0]!.name).toBe("Kant Critique"); + expect(res.markers[0]!.date).toBe("1781-05-01"); + expect(res.markers[0]!.line).toBe(1); + expect(res.malformed).toHaveLength(0); + }); + + test("reports a missing-comma marker as malformed and skips it", () => { + const res = parseCitations("[Source: NoComma 2026-01-01]\n"); + expect(res.markers).toHaveLength(0); + expect(res.malformed).toHaveLength(1); + expect(res.malformed[0]!.reason).toMatch(/comma|date/i); + }); + + test("reports a bad-date-shape marker as malformed and skips it", () => { + const res = parseCitations("[Source: Foo, 2026-1-1]\n[Source: Bar, 2026-13-40]\n"); + expect(res.markers).toHaveLength(0); + expect(res.malformed).toHaveLength(2); + }); + + test("handles multiple markers across lines with correct line numbers", () => { + const res = parseCitations("line one\n[Source: A, 2020-01-02]\n\n[Source: B, 2021-03-04]\n"); + expect(res.markers.map((m) => [m.name, m.date, m.line])).toEqual([ + ["A", "2020-01-02", 2], + ["B", "2021-03-04", 4], + ]); + }); + + test("notes without markers produce nothing", () => { + const res = parseCitations("Just prose, no markers here.\n"); + expect(res.markers).toHaveLength(0); + expect(res.malformed).toHaveLength(0); + }); + + test("date parsing is structural, not natural-language", () => { + // A localized month name must NOT be recognised as a date. + const res = parseCitations("[Source: Foo, May 1 2026]\n"); + expect(res.markers).toHaveLength(0); + expect(res.malformed).toHaveLength(1); + }); +}); + +describe("normalizeCitationName", () => { + test("folds case and collapses whitespace for dedup", () => { + expect(normalizeCitationName(" Kant Critique ")).toBe( + normalizeCitationName("kant critique"), + ); + }); +}); + +describe("scanCitations promotion", () => { + test("a well-formed marker becomes a dated provenance event on the timeline", () => { + writeMd("Notes/a.md", "Claim. [Source: Origin of Species, 1859-11-24]\n"); + const res = scanCitations(tmp, { agent: "tester" }); + expect(res.found).toBe(1); + expect(res.promoted).toBe(1); + expect(res.deduped).toBe(0); + + const events = sourceCitationEvents(tmp); + expect(events).toHaveLength(1); + // Dated at the citation date, not the scan time. + expect(events[0]!.at).toBe("1859-11-24T00:00:00Z"); + expect(events[0]!.source.path).toContain("1859-11-24"); + }); + + test("re-scanning does not duplicate (dedup on normalized name + date)", () => { + writeMd("Notes/a.md", "[Source: Origin of Species, 1859-11-24]\n"); + scanCitations(tmp, { agent: "tester" }); + const second = scanCitations(tmp, { agent: "tester" }); + expect(second.promoted).toBe(0); + expect(second.deduped).toBe(1); + expect(sourceCitationEvents(tmp)).toHaveLength(1); + }); + + test("normalized-name variants dedup against an already-logged event", () => { + writeMd("Notes/a.md", "[Source: Origin of Species, 1859-11-24]\n"); + writeMd("Notes/b.md", "[Source: origin of species , 1859-11-24]\n"); + const res = scanCitations(tmp, { agent: "tester" }); + expect(res.found).toBe(2); + expect(res.promoted).toBe(1); + expect(res.deduped).toBe(1); + expect(sourceCitationEvents(tmp)).toHaveLength(1); + }); + + test("malformed markers are reported explicitly and skipped", () => { + writeMd("Notes/a.md", "[Source: Bad, 2026-13-40]\n[Source: NoComma 2026-01-01]\n"); + const res = scanCitations(tmp, { agent: "tester" }); + expect(res.promoted).toBe(0); + expect(res.malformed).toBe(2); + expect(res.malformedMarkers).toHaveLength(2); + expect(sourceCitationEvents(tmp)).toHaveLength(0); + }); + + test("notes without markers produce no events", () => { + writeMd("Notes/a.md", "No provenance markers at all.\n"); + const res = scanCitations(tmp, { agent: "tester" }); + expect(res.found).toBe(0); + expect(res.promoted).toBe(0); + expect(sourceCitationEvents(tmp)).toHaveLength(0); + }); + + test("dry run reports promotions without writing events", () => { + writeMd("Notes/a.md", "[Source: Origin, 1859-11-24]\n"); + const res = scanCitations(tmp, { agent: "tester", dryRun: true }); + expect(res.promoted).toBe(1); + expect(sourceCitationEvents(tmp)).toHaveLength(0); + }); + + test("distinct dates for one source promote separate events", () => { + writeMd("Notes/a.md", "[Source: Journal, 2020-01-01]\n[Source: Journal, 2021-01-01]\n"); + const res = scanCitations(tmp, { agent: "tester" }); + expect(res.promoted).toBe(2); + expect(sourceCitationEvents(tmp)).toHaveLength(2); + }); +}); From 0a83677ee5fbb981855c02a2fcc160b2a586396f Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 21:39:49 +0000 Subject: [PATCH 09/14] feat(search): configurable FTS tokenizer language and diacritic rules Assemble the `chunk_fts` FTS5 tokenizer clause from validated config keys instead of a hardcoded literal, so operators can tune diacritic folding and layer Porter stemming without patching the schema. - New `buildFtsTokenize` in schema.ts composes the clause from an allow-list (`search_fts_diacritics` 0|1|2, `search_fts_stemmer` none|porter); an out-of-range value throws a typed SearchError("INVALID_INPUT") listing the allowed values before it can reach the DDL. - Migrations that create `chunk_fts` (v1, v2, v5) take the clause via a MigrationContext threaded through applyMigrations; unset config yields the historical `unicode61 remove_diacritics 2` byte-identically. The CJK `chunk_trigram` shadow index keeps its `trigram` tokenizer. - resolveSearchConfig resolves and validates the keys once (ResolvedSearchConfig.ftsTokenize); Store.open passes the clause to applyMigrations. No implicit reindex: the clause only takes effect on a fresh build, documented in the CLI reference. Co-Authored-By: Claude Fable 5 --- docs/cli-reference.md | 9 +++ src/core/search/index.ts | 16 ++++ src/core/search/schema.ts | 102 +++++++++++++++++++++--- src/core/search/store.ts | 2 +- src/core/search/types.ts | 10 +++ tests/core/search/fts-tokenizer.test.ts | 100 +++++++++++++++++++++++ tests/core/search/store.test.ts | 3 +- tests/core/search/store.vec.test.ts | 2 + tests/helpers/search-fixtures.ts | 2 + 9 files changed, 231 insertions(+), 15 deletions(-) create mode 100644 tests/core/search/fts-tokenizer.test.ts diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 51a15d4a..3a42e994 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -353,6 +353,15 @@ estimated spend exceeds it unless `--force-cost`. `search_fusion_mode` lanes by reciprocal rank (`search_rrf_k`, default 60); `linear` keeps ranking bit-identical. +FTS tokenizer (since v1.34.0): `search_fts_diacritics` (default `2`; also +`0`/`1`) sets the `unicode61 remove_diacritics` rule, and +`search_fts_stemmer` (default `none`; also `porter`) layers Porter +stemming over it. Unset keys keep the historical +`unicode61 remove_diacritics 2` clause byte-identically. An out-of-range +value is rejected loudly; the CJK trigram prefilter is unaffected. +Changing either key only takes effect after `o2b search reindex` — there +is no implicit reindex. + Typed relations participate in ranking (relation polarity): a page whose frontmatter declares `superseded_by:` is demoted when it matches and its successor is boosted or pulled in, `contradicts:` surfaces warning-style diff --git a/src/core/search/index.ts b/src/core/search/index.ts index 15c8491f..461cda0b 100644 --- a/src/core/search/index.ts +++ b/src/core/search/index.ts @@ -12,6 +12,7 @@ import { } from "../validate.ts"; import { resolveVaultScope } from "../vault-scope/index.ts"; import { resolveIndexPath } from "./paths.ts"; +import { buildFtsTokenize } from "./schema.ts"; import { isFusionMode, DEFAULT_RRF_K } from "./fusion.ts"; import { loadProviderRegistry, @@ -787,6 +788,20 @@ export function resolveSearchConfig(opts: { DEFAULTS.resumeReindex, "search_resume_reindex", ); + // Configurable FTS tokenizer (t_618f7211). Both keys default to null, + // which yields the byte-identical `unicode61 remove_diacritics 2` + // clause; an out-of-range value throws SearchError("INVALID_INPUT") + // here rather than reaching the DDL. Changing either key takes effect + // only after `o2b search reindex`; there is no implicit reindex. + const ftsTokenize = buildFtsTokenize({ + diacritics: rawSetting( + env, + config, + "OPEN_SECOND_BRAIN_SEARCH_FTS_DIACRITICS", + "search_fts_diacritics", + ), + stemmer: rawSetting(env, config, "OPEN_SECOND_BRAIN_SEARCH_FTS_STEMMER", "search_fts_stemmer"), + }); const recall: ResolvedRecallConfig = Object.freeze({ mmrLambda, maxHops, @@ -829,6 +844,7 @@ export function resolveSearchConfig(opts: { rerank, shutdownGraceMs: shutdownGraceSeconds * 1000, resumeReindex, + ftsTokenize, }); if (!opts.overrides) { diff --git a/src/core/search/schema.ts b/src/core/search/schema.ts index 7f14a672..8c1c0928 100644 --- a/src/core/search/schema.ts +++ b/src/core/search/schema.ts @@ -32,7 +32,68 @@ export function documentBasename(path: string): string { return name.endsWith(".md") ? name.slice(0, -".md".length) : name; } -const DDL_V1 = ` +/** + * The historical, byte-stable FTS5 tokenizer clause. When no tokenizer + * config is set, {@link buildFtsTokenize} returns exactly this string so + * the generated `chunk_fts` schema is identical to every prior release. + */ +export const DEFAULT_FTS_TOKENIZE = "unicode61 remove_diacritics 2"; + +/** Allowed `remove_diacritics` values for the unicode61 base tokenizer. */ +export const FTS_DIACRITICS_ALLOWED: ReadonlyArray = Object.freeze(["0", "1", "2"]); + +/** Allowed stemmer selectors layered over the unicode61 base tokenizer. */ +export const FTS_STEMMER_ALLOWED: ReadonlyArray = Object.freeze(["none", "porter"]); + +/** Validated inputs for {@link buildFtsTokenize}. A null/absent value uses the default. */ +export interface FtsTokenizerOptions { + /** `remove_diacritics` rule: "0" | "1" | "2". Absent/null = "2". */ + readonly diacritics?: string | null; + /** Language stemmer: "none" | "porter". Absent/null = "none". */ + readonly stemmer?: string | null; +} + +/** + * Assemble the FTS5 tokenizer clause from validated config options. The + * clause is composed structurally from a fixed allow-list of FTS5 + * tokenizer options (never a natural-language word list), so an operator + * can only ever produce a valid SQLite tokenizer directive; an + * out-of-range value rejects with a typed `SearchError` listing the + * allowed values rather than reaching the DDL. + * + * With no options set the return value is byte-identical to + * {@link DEFAULT_FTS_TOKENIZE}. Changing it takes effect only on the next + * `o2b search reindex`; there is no implicit reindex. + */ +export function buildFtsTokenize(opts: FtsTokenizerOptions): string { + const diacritics = opts.diacritics ?? "2"; + if (!FTS_DIACRITICS_ALLOWED.includes(diacritics)) { + throw new SearchError( + "INVALID_INPUT", + `search_fts_diacritics must be one of ${FTS_DIACRITICS_ALLOWED.join(", ")}; got '${diacritics}'`, + ); + } + const stemmer = opts.stemmer ?? "none"; + if (!FTS_STEMMER_ALLOWED.includes(stemmer)) { + throw new SearchError( + "INVALID_INPUT", + `search_fts_stemmer must be one of ${FTS_STEMMER_ALLOWED.join(", ")}; got '${stemmer}'`, + ); + } + const base = `unicode61 remove_diacritics ${diacritics}`; + // The porter stemmer WRAPS a base tokenizer in FTS5 syntax, so it + // prefixes the unicode61 directive rather than replacing it. + return stemmer === "porter" ? `porter ${base}` : base; +} + +/** Context threaded into every migration's `up`. */ +export interface MigrationContext { + /** FTS5 tokenizer clause for the searchable `chunk_fts` table. */ + readonly ftsTokenize: string; +} + +function ddlV1(ftsTokenize: string): string { + return ` CREATE TABLE IF NOT EXISTS documents ( id INTEGER PRIMARY KEY, path TEXT NOT NULL UNIQUE, @@ -66,7 +127,7 @@ CREATE VIRTUAL TABLE IF NOT EXISTS chunk_fts USING fts5( content, content='chunks', content_rowid='id', - tokenize='unicode61 remove_diacritics 2' + tokenize='${ftsTokenize}' ); CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN @@ -115,10 +176,11 @@ CREATE TABLE IF NOT EXISTS index_state ( updated_at TEXT NOT NULL ); `; +} interface Migration { readonly version: number; - readonly up: (db: Database) => void; + readonly up: (db: Database, ctx: MigrationContext) => void; } /** @@ -148,7 +210,8 @@ CREATE INDEX IF NOT EXISTS idx_chunk_entities_entity ON chunk_entities(entity); // FTS rebuild is split from the column add because `ALTER TABLE ADD // COLUMN` is not idempotent; the column add is guarded in `up()`. -const DDL_V2_FTS = ` +function ddlV2Fts(ftsTokenize: string): string { + return ` DROP TRIGGER IF EXISTS chunks_ai; DROP TRIGGER IF EXISTS chunks_ad; DROP TRIGGER IF EXISTS chunks_au; @@ -159,7 +222,7 @@ CREATE VIRTUAL TABLE chunk_fts USING fts5( heading_path, content='chunks', content_rowid='id', - tokenize='unicode61 remove_diacritics 2' + tokenize='${ftsTokenize}' ); CREATE TRIGGER chunks_ai AFTER INSERT ON chunks BEGIN @@ -179,23 +242,24 @@ END; INSERT INTO chunk_fts(chunk_fts) VALUES('rebuild'); `; +} export const MIGRATIONS: ReadonlyArray = Object.freeze([ { version: 1, - up(db) { - db.exec(DDL_V1); + up(db, ctx) { + db.exec(ddlV1(ctx.ftsTokenize)); }, }, { version: 2, - up(db) { + up(db, ctx) { db.exec(DDL_V2_ENTITIES); const cols = db.query<{ name: string }, []>("PRAGMA table_info(chunks)").all(); if (!cols.some((c) => c.name === "heading_path")) { db.exec("ALTER TABLE chunks ADD COLUMN heading_path TEXT NOT NULL DEFAULT ''"); } - db.exec(DDL_V2_FTS); + db.exec(ddlV2Fts(ctx.ftsTokenize)); }, }, { @@ -240,7 +304,7 @@ export const MIGRATIONS: ReadonlyArray = Object.freeze([ // an expanded FTS shadow column. Existing rows default to their // current content until a reindex computes CJK token expansions. version: 5, - up(db) { + up(db, ctx) { const cols = db.query<{ name: string }, []>("PRAGMA table_info(chunks)").all(); if (!cols.some((c) => c.name === "fts_content")) { db.exec("ALTER TABLE chunks ADD COLUMN fts_content TEXT NOT NULL DEFAULT ''"); @@ -257,7 +321,7 @@ export const MIGRATIONS: ReadonlyArray = Object.freeze([ heading_path, content='chunks', content_rowid='id', - tokenize='unicode61 remove_diacritics 2' + tokenize='${ctx.ftsTokenize}' ); CREATE TRIGGER chunks_ai AFTER INSERT ON chunks BEGIN @@ -455,6 +519,17 @@ function setSchemaVersion(db: Database, version: number): void { ); } +/** Options for {@link applyMigrations}. */ +export interface ApplyMigrationsOptions { + /** + * FTS5 tokenizer clause for the searchable `chunk_fts` table. Absent = + * {@link DEFAULT_FTS_TOKENIZE}, which keeps the generated schema + * byte-identical to prior releases. Only takes effect on a fresh build + * (reindex); an already-migrated index is not rewritten. + */ + readonly ftsTokenize?: string; +} + /** * Apply every pending migration in a single transaction. On a fresh * database (no `index_state`), v1 creates the entire schema. On an @@ -462,7 +537,8 @@ function setSchemaVersion(db: Database, version: number): void { * * Returns the version after migration. */ -export function applyMigrations(db: Database): number { +export function applyMigrations(db: Database, opts: ApplyMigrationsOptions = {}): number { + const ctx: MigrationContext = { ftsTokenize: opts.ftsTokenize ?? DEFAULT_FTS_TOKENIZE }; const hasIndexState = db .query<{ name: string }, []>( "SELECT name FROM sqlite_master WHERE type='table' AND name='index_state'", @@ -485,7 +561,7 @@ export function applyMigrations(db: Database): number { try { for (const migration of MIGRATIONS) { if (migration.version <= current) continue; - migration.up(db); + migration.up(db, ctx); setSchemaVersion(db, migration.version); } db.exec("COMMIT"); diff --git a/src/core/search/store.ts b/src/core/search/store.ts index 58779044..8633ffce 100644 --- a/src/core/search/store.ts +++ b/src/core/search/store.ts @@ -463,7 +463,7 @@ export class Store { try { applyPragmas(db); - applyMigrations(db); + applyMigrations(db, { ftsTokenize: config.ftsTokenize }); ensureFts5(db); const vecLoaded = loadVec && tryLoadVecExtension(db); const store = new Store(db, config, vecLoaded, release); diff --git a/src/core/search/types.ts b/src/core/search/types.ts index 4382e663..b3a39523 100644 --- a/src/core/search/types.ts +++ b/src/core/search/types.ts @@ -1006,4 +1006,14 @@ export interface ResolvedSearchConfig { * gated on a signature marker, so a drifted staging DB is rebuilt. */ readonly resumeReindex: boolean; + /** + * FTS5 tokenizer clause for the searchable `chunk_fts` table (Q1 of the + * search config, t_618f7211), assembled from validated + * `search_fts_diacritics` / `search_fts_stemmer` config keys. Absent + * config yields the historical `unicode61 remove_diacritics 2` + * byte-identically. Applied only when the index is (re)built, so a + * change requires an explicit `o2b search reindex`; the CJK trigram + * shadow index is untouched by this clause. + */ + readonly ftsTokenize: string; } diff --git a/tests/core/search/fts-tokenizer.test.ts b/tests/core/search/fts-tokenizer.test.ts new file mode 100644 index 00000000..f95156ac --- /dev/null +++ b/tests/core/search/fts-tokenizer.test.ts @@ -0,0 +1,100 @@ +/** + * Task Q2 (t_618f7211): configurable FTS tokenizer language and + * diacritic rules. + * + * Acceptance coverage: + * - with no config the generated schema keeps + * `unicode61 remove_diacritics 2` byte-identically; + * - a valid config changes the tokenizer clause on the materialized + * chunk_fts table (after reindex); + * - invalid tokenizer options reject with a typed error listing the + * allowed values; + * - the CJK trigram path is untouched (chunk_trigram stays `trigram`); + * - no implicit reindex (buildFtsTokenize is pure; changing config + * alone rewrites nothing). + */ + +import { describe, expect, test } from "bun:test"; +import { Database } from "bun:sqlite"; + +import { + applyMigrations, + buildFtsTokenize, + DEFAULT_FTS_TOKENIZE, +} from "../../../src/core/search/schema.ts"; +import { SearchError } from "../../../src/core/search/types.ts"; + +function ftsSql(db: Database, table: string): string { + const row = db + .query<{ sql: string }, [string]>("SELECT sql FROM sqlite_master WHERE name = ?") + .get(table); + return row?.sql ?? ""; +} + +describe("buildFtsTokenize", () => { + test("no config yields the byte-identical default clause", () => { + expect(buildFtsTokenize({})).toBe("unicode61 remove_diacritics 2"); + expect(buildFtsTokenize({})).toBe(DEFAULT_FTS_TOKENIZE); + expect(buildFtsTokenize({ diacritics: null, stemmer: null })).toBe(DEFAULT_FTS_TOKENIZE); + }); + + test("diacritics option changes the remove_diacritics rule", () => { + expect(buildFtsTokenize({ diacritics: "0" })).toBe("unicode61 remove_diacritics 0"); + expect(buildFtsTokenize({ diacritics: "1" })).toBe("unicode61 remove_diacritics 1"); + }); + + test("porter stemmer wraps unicode61 (language stemming)", () => { + expect(buildFtsTokenize({ stemmer: "porter" })).toBe("porter unicode61 remove_diacritics 2"); + expect(buildFtsTokenize({ stemmer: "porter", diacritics: "0" })).toBe( + "porter unicode61 remove_diacritics 0", + ); + }); + + test("invalid diacritics rejects with a typed error listing allowed values", () => { + let err: unknown; + try { + buildFtsTokenize({ diacritics: "3" }); + } catch (e) { + err = e; + } + expect(err).toBeInstanceOf(SearchError); + expect((err as SearchError).code).toBe("INVALID_INPUT"); + expect((err as SearchError).message).toMatch(/0.*1.*2/); + }); + + test("invalid stemmer rejects with a typed error listing allowed values", () => { + let err: unknown; + try { + buildFtsTokenize({ stemmer: "snowball" }); + } catch (e) { + err = e; + } + expect(err).toBeInstanceOf(SearchError); + expect((err as SearchError).code).toBe("INVALID_INPUT"); + expect((err as SearchError).message).toMatch(/none.*porter|porter.*none/); + }); +}); + +describe("applyMigrations tokenizer wiring", () => { + test("default schema keeps unicode61 remove_diacritics 2 byte-identically", () => { + const db = new Database(":memory:"); + applyMigrations(db); + expect(ftsSql(db, "chunk_fts")).toContain("tokenize='unicode61 remove_diacritics 2'"); + db.close(); + }); + + test("a configured clause is applied to the materialized chunk_fts table", () => { + const db = new Database(":memory:"); + applyMigrations(db, { ftsTokenize: buildFtsTokenize({ stemmer: "porter" }) }); + expect(ftsSql(db, "chunk_fts")).toContain("tokenize='porter unicode61 remove_diacritics 2'"); + db.close(); + }); + + test("the CJK trigram path stays trigram regardless of the tokenizer config", () => { + const db = new Database(":memory:"); + applyMigrations(db, { ftsTokenize: buildFtsTokenize({ stemmer: "porter", diacritics: "0" }) }); + expect(ftsSql(db, "chunk_trigram")).toContain("tokenize='trigram'"); + expect(ftsSql(db, "chunk_trigram")).not.toContain("porter"); + db.close(); + }); +}); diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.ts index fa2262a3..04d84216 100644 --- a/tests/core/search/store.test.ts +++ b/tests/core/search/store.test.ts @@ -4,7 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { Store } from "../../../src/core/search/store.ts"; -import { LATEST_SCHEMA_VERSION } from "../../../src/core/search/schema.ts"; +import { DEFAULT_FTS_TOKENIZE, LATEST_SCHEMA_VERSION } from "../../../src/core/search/schema.ts"; import { SearchError } from "../../../src/core/search/types.ts"; import type { ResolvedSearchConfig, @@ -77,6 +77,7 @@ function makeConfig(overrides?: Partial): ResolvedSearchCo }), shutdownGraceMs: 5_000, resumeReindex: false, + ftsTokenize: DEFAULT_FTS_TOKENIZE, ...overrides, }); } diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.ts index 0753e2fd..0f4a9c3c 100644 --- a/tests/core/search/store.vec.test.ts +++ b/tests/core/search/store.vec.test.ts @@ -4,6 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { Store } from "../../../src/core/search/store.ts"; +import { DEFAULT_FTS_TOKENIZE } from "../../../src/core/search/schema.ts"; import type { ResolvedSearchConfig, ResolvedEmbeddingConfig, @@ -80,6 +81,7 @@ function semanticConfig( }), shutdownGraceMs: 5_000, resumeReindex: false, + ftsTokenize: DEFAULT_FTS_TOKENIZE, ...overrides, }); } diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.ts index 56704e17..db4419de 100644 --- a/tests/helpers/search-fixtures.ts +++ b/tests/helpers/search-fixtures.ts @@ -16,6 +16,7 @@ import type { ResolvedEmbeddingConfig, ResolvedRerankConfig, } from "../../src/core/search/types.ts"; +import { DEFAULT_FTS_TOKENIZE } from "../../src/core/search/schema.ts"; export function createTempVault(prefix: string): { vault: string; @@ -157,5 +158,6 @@ export function makeConfig(opts: { }), shutdownGraceMs: 5_000, resumeReindex: false, + ftsTokenize: DEFAULT_FTS_TOKENIZE, }); } From 6b2ff62fbe51d07a21f58886a8756058f91c8f9c Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 21:50:08 +0000 Subject: [PATCH 10/14] feat(search): graph-degree cardinality predicates in filter DSL Add count predicates over a note's backlinks/outlinks to the search filter DSL, so operators and agents can select orphans (`backlinks=0`), hubs (`outlinks>=N`), and any `= != > >= < <=` comparison. - graph-index.ts gains directed in/out degree maps on the memoized snapshot plus a `degreeForPath` accessor; a path with no resolved edges (or an unknown path) reads as an orphan. - property-filter.ts gains `parseDegreePredicate` (typed SearchError("INVALID_INPUT") on bad syntax, listing allowed fields and operators) and a pure `applyDegreeFilters` (dependency-injected degree lookup; empty list is a byte-identical pass-through). - search.ts applies the predicates as a post-rank phase backed by the link-graph snapshot; result-filters.ts wires the store. CLI `--degree` and the MCP brain_search `degree` param feed the same predicate list. - Queries without degree predicates are byte-identical; the MCP tool count stays 102 (a new param on the existing tool, not a new tool). Co-Authored-By: Claude Fable 5 --- docs/cli-reference.md | 2 + src/cli/search.ts | 17 +++ src/core/brain/link-graph/graph-index.ts | 49 ++++++++ src/core/search/property-filter.ts | 100 +++++++++++++++ src/core/search/result-filters.ts | 22 +++- src/core/search/search.ts | 19 ++- src/core/search/types.ts | 9 ++ src/mcp/search-tools.ts | 34 ++++++ tests/core/brain/graph-index.test.ts | 33 ++++- .../core/search/degree-filter-search.test.ts | 66 ++++++++++ tests/core/search/degree-filter.test.ts | 115 ++++++++++++++++++ 11 files changed, 461 insertions(+), 5 deletions(-) create mode 100644 tests/core/search/degree-filter-search.test.ts create mode 100644 tests/core/search/degree-filter.test.ts diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 3a42e994..64a2f4b3 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -311,6 +311,8 @@ o2b partner codegraph report Resolve the in-scope code project and report the c o2b search "" Hybrid full-text + semantic search across the vault --property type=decision --property status=open filters on frontmatter scalars (post-FTS phase) + --degree backlinks=0 / --degree outlinks>=5 filter by graph + cardinality (orphans, hubs; ops = != > >= < <=, ANDed) --query-doc '' separates intent/lex/vec/hyde recall lanes --evidence-pack adds matched/missing term diagnostics, abstention text, IDF-weighted coverage, per-token union records, and a completeness verdict diff --git a/src/cli/search.ts b/src/cli/search.ts index 2b923b12..51884486 100644 --- a/src/cli/search.ts +++ b/src/cli/search.ts @@ -56,6 +56,7 @@ import { SEARCH_LIMIT_MIN, SEARCH_LIMIT_MAX, } from "../core/search/index.ts"; +import { parseDegreePredicate, type DegreePredicate } from "../core/search/property-filter.ts"; import type { IndexCheckReport, IndexProgressEvent, @@ -620,6 +621,7 @@ async function cmdSearchQuery(argv: ReadonlyArray): Promise { "semantic-weight": { type: "string" }, "auto-refresh": { type: "boolean" }, property: { type: "string-array" }, + degree: { type: "string-array" }, visibility: { type: "string-array" }, "query-doc": { type: "string" }, expand: { type: "boolean" }, @@ -680,6 +682,7 @@ async function cmdSearchQuery(argv: ReadonlyArray): Promise { flags["semantic"] === true ? true : flags["keyword-only"] === true ? false : undefined; const properties = parsePropertyFlags(flags["property"] as string[] | undefined); + const degreeFilters = parseDegreeFlags(flags["degree"] as string[] | undefined); const visibility = flags["visibility"] as string[] | undefined; const searchOpts = { @@ -689,6 +692,7 @@ async function cmdSearchQuery(argv: ReadonlyArray): Promise { keywordOnly: flags["keyword-only"] === true, pathPrefix: typeof flags["path"] === "string" ? (flags["path"] as string) : undefined, ...(properties !== undefined ? { properties } : {}), + ...(degreeFilters !== undefined ? { degreeFilters } : {}), ...(visibility !== undefined && visibility.length > 0 ? { visibility } : {}), ...(structuredQuery !== undefined ? { structuredQuery } : {}), ...(flags["expand"] === true ? { expand: true } : {}), @@ -755,6 +759,19 @@ function parsePropertyFlags( return frozen; } +/** + * Parse the repeatable `--degree ` flag into the + * `degreeFilters` predicate list that `search()` consumes, e.g. + * `--degree backlinks=0` (orphans) or `--degree outlinks>=5` (hubs). + * Invalid syntax throws a typed `SearchError` from + * {@link parseDegreePredicate}, surfaced by the search verb's error + * handler as exit 2. + */ +function parseDegreeFlags(raw: ReadonlyArray | undefined): DegreePredicate[] | undefined { + if (!raw || raw.length === 0) return undefined; + return raw.map((entry) => parseDegreePredicate(entry)); +} + function jsonForOutcome(o: SearchOutcome): unknown { return { results: o.results.map((r) => ({ diff --git a/src/core/brain/link-graph/graph-index.ts b/src/core/brain/link-graph/graph-index.ts index a22bee25..44bf479a 100644 --- a/src/core/brain/link-graph/graph-index.ts +++ b/src/core/brain/link-graph/graph-index.ts @@ -32,6 +32,20 @@ export interface GraphSnapshot { readonly adjacency: ReadonlyMap>; /** documentId -> neighbour count (degree). Mirrors {@link adjacency}. */ readonly degree: ReadonlyMap; + /** + * documentId -> out-link count: distinct resolved documents this note + * links to (self-loops dropped). Only documents with at least one + * out-link appear; absence means zero. Feeds the `outlinks` filter + * predicate. + */ + readonly outDegree: ReadonlyMap; + /** + * documentId -> back-link count: distinct resolved documents that link + * to this note (self-loops dropped). Only documents with at least one + * back-link appear; absence means zero. Feeds the `backlinks` filter + * predicate. + */ + readonly inDegree: ReadonlyMap; /** Linked node ids in ascending order (the deterministic sweep order). */ readonly nodesSorted: ReadonlyArray; /** Count of distinct undirected edges. */ @@ -66,6 +80,11 @@ function buildGraphSnapshot(store: Store): GraphSnapshot { } const adjacency = new Map>(); + // Directed neighbour sets, kept distinct so a note that links to the + // same target twice counts it once. Backlinks read `inNeighbours`, + // out-links read `outNeighbours`. + const outNeighbours = new Map>(); + const inNeighbours = new Map>(); let edgeCount = 0; for (const { source, target } of store.resolvedDocLinkPairs()) { if (source === target || !pathById.has(source) || !pathById.has(target)) continue; @@ -77,10 +96,16 @@ function buildGraphSnapshot(store: Store): GraphSnapshot { } const back = adjacency.get(target) ?? adjacency.set(target, new Set()).get(target)!; back.add(source); + (outNeighbours.get(source) ?? outNeighbours.set(source, new Set()).get(source)!).add(target); + (inNeighbours.get(target) ?? inNeighbours.set(target, new Set()).get(target)!).add(source); } const degree = new Map(); for (const [node, neighbours] of adjacency) degree.set(node, neighbours.size); + const outDegree = new Map(); + for (const [node, neighbours] of outNeighbours) outDegree.set(node, neighbours.size); + const inDegree = new Map(); + for (const [node, neighbours] of inNeighbours) inDegree.set(node, neighbours.size); const nodesSorted = [...adjacency.keys()].toSorted((a, b) => a - b); @@ -90,11 +115,35 @@ function buildGraphSnapshot(store: Store): GraphSnapshot { idByPath, adjacency, degree, + outDegree, + inDegree, nodesSorted, edgeCount, }); } +/** Directed degree of one note: distinct back-links and out-links. */ +export interface NoteDegree { + /** Distinct resolved documents that link TO this note. */ + readonly backlinks: number; + /** Distinct resolved documents this note links to. */ + readonly outlinks: number; +} + +/** + * Look up a note's directed degree by vault-relative path. A path with no + * resolved edges (or an unknown path) is an orphan: `{ backlinks: 0, + * outlinks: 0 }`. Pure over the snapshot; no I/O. + */ +export function degreeForPath(snapshot: GraphSnapshot, path: string): NoteDegree { + const id = snapshot.idByPath.get(path); + if (id === undefined) return { backlinks: 0, outlinks: 0 }; + return { + backlinks: snapshot.inDegree.get(id) ?? 0, + outlinks: snapshot.outDegree.get(id) ?? 0, + }; +} + export interface GraphDegreeEntry { readonly path: string; readonly degree: number; diff --git a/src/core/search/property-filter.ts b/src/core/search/property-filter.ts index d47d2f34..de57ec47 100644 --- a/src/core/search/property-filter.ts +++ b/src/core/search/property-filter.ts @@ -17,9 +17,109 @@ * reader, tests can pass an in-memory map. */ +import { SearchError } from "./types.ts"; + export type PropertyFilterMap = ReadonlyMap>; export type PropertyFrontmatterReader = (path: string) => Record | null; +// ───────────────────────────────────────────────────────────────────────────── +// Graph-degree cardinality predicates (t_9bee8f0b) +// ───────────────────────────────────────────────────────────────────────────── + +/** Degree field a predicate ranges over. */ +export type DegreeField = "backlinks" | "outlinks"; + +/** Comparison operator for a degree predicate. */ +export type DegreeOp = "=" | "!=" | ">" | ">=" | "<" | "<="; + +/** A parsed `` graph-degree predicate. */ +export interface DegreePredicate { + readonly field: DegreeField; + readonly op: DegreeOp; + /** Non-negative integer count the field is compared against. */ + readonly value: number; +} + +/** A note's directed degree: distinct back-links and out-links. */ +export interface DegreeLookupResult { + readonly backlinks: number; + readonly outlinks: number; +} + +/** Resolves a result path to its directed degree counts. */ +export type DegreeLookup = (path: string) => DegreeLookupResult; + +const DEGREE_FIELDS: ReadonlyArray = Object.freeze(["backlinks", "outlinks"]); +const DEGREE_OPS: ReadonlyArray = Object.freeze(["=", "!=", ">", ">=", "<", "<="]); +// Longer operators first so `>=` is not mis-split as `>` + `=`. +const DEGREE_PREDICATE_RE = /^(backlinks|outlinks)\s*(!=|>=|<=|=|>|<)\s*(\d+)$/; + +/** + * Parse one `` graph-degree predicate, e.g. + * `backlinks=0` (orphans) or `outlinks>=5` (hubs). The count must be a + * non-negative integer. Invalid syntax rejects with a typed + * `SearchError("INVALID_INPUT")` naming the allowed fields and operators + * rather than silently ignoring the predicate. + */ +export function parseDegreePredicate(raw: string): DegreePredicate { + const trimmed = raw.trim(); + const m = DEGREE_PREDICATE_RE.exec(trimmed); + if (m === null) { + throw new SearchError( + "INVALID_INPUT", + `invalid degree predicate '${raw}'; expected where ` + + `field is one of ${DEGREE_FIELDS.join(", ")}, op is one of ${DEGREE_OPS.join(", ")}, ` + + "and count is a non-negative integer (e.g. backlinks=0, outlinks>=5)", + ); + } + return Object.freeze({ + field: m[1] as DegreeField, + op: m[2] as DegreeOp, + value: Number(m[3]), + }); +} + +function compareDegree(actual: number, op: DegreeOp, value: number): boolean { + switch (op) { + case "=": + return actual === value; + case "!=": + return actual !== value; + case ">": + return actual > value; + case ">=": + return actual >= value; + case "<": + return actual < value; + case "<=": + return actual <= value; + } +} + +/** + * Drop rows whose backlink/outlink counts do not satisfy every predicate + * (AND across predicates). The degree lookup is dependency-injected so + * this helper stays pure; the search orchestrator wires the live graph + * snapshot. An empty predicate list is a byte-identical pass-through. + */ +export function applyDegreeFilters( + results: ReadonlyArray, + predicates: ReadonlyArray, + degreeOf: DegreeLookup, +): ReadonlyArray { + if (predicates.length === 0) return Object.freeze([...results]) as ReadonlyArray; + const out: T[] = []; + for (const row of results) { + const degree = degreeOf(row.path); + const ok = predicates.every((p) => { + const actual = p.field === "backlinks" ? degree.backlinks : degree.outlinks; + return compareDegree(actual, p.op, p.value); + }); + if (ok) out.push(row); + } + return Object.freeze(out) as ReadonlyArray; +} + export function filterByProperties( results: ReadonlyArray, filters: PropertyFilterMap, diff --git a/src/core/search/result-filters.ts b/src/core/search/result-filters.ts index 3a864576..a7646421 100644 --- a/src/core/search/result-filters.ts +++ b/src/core/search/result-filters.ts @@ -12,7 +12,9 @@ import { parseFrontmatter } from "../vault.ts"; import type { FrontmatterMap } from "../types.ts"; import { isVisible, pageVisibility } from "../graph/visibility.ts"; import { isOwnerVisible, pageOwner } from "../graph/agent-scope.ts"; -import { filterByProperties } from "./property-filter.ts"; +import { applyDegreeFilters, filterByProperties, type DegreePredicate } from "./property-filter.ts"; +import { degreeForPath, getGraphSnapshot } from "../brain/link-graph/graph-index.ts"; +import type { Store } from "./store.ts"; import { deriveTrust } from "./enrich.ts"; import { isTerminalStatus } from "./evidence-pack.ts"; import { isTombstoned } from "../brain/lifecycle/tombstone.ts"; @@ -130,6 +132,24 @@ export function attachTrustMetadata( }); } +/** + * Graph-degree cardinality filter (t_9bee8f0b): drop rows whose + * backlink/outlink counts do not satisfy every predicate. Degree data + * comes from the memoized link-graph snapshot, so a repeat query against + * an unchanged index does no graph rebuild. An empty predicate list is a + * byte-identical pass-through (the caller gates on that, but this stays + * safe if called directly). + */ +export function applyDegreeFilter( + ranked: ReadonlyArray, + predicates: ReadonlyArray, + store: Store, +): ReadonlyArray { + if (predicates.length === 0) return ranked; + const snapshot = getGraphSnapshot(store); + return applyDegreeFilters(ranked, predicates, (path) => degreeForPath(snapshot, path)); +} + export function applyVisibilityScope( ranked: ReadonlyArray, scope: ReadonlySet, diff --git a/src/core/search/search.ts b/src/core/search/search.ts index fe7b6939..185872dc 100644 --- a/src/core/search/search.ts +++ b/src/core/search/search.ts @@ -77,6 +77,7 @@ import { resolveSemanticPolicy, runSemanticPhase, semanticPoolSize } from "./sem import { toSearchCard } from "./cards.ts"; import { applyAgentScope, + applyDegreeFilter, applyPropertyFilter, applyStatusFilter, applyVisibilityScope, @@ -737,6 +738,11 @@ export async function search( // candidates to the filter and surface zero results even when // matches exist deeper in the rank. const hasPropertyFilter = opts.properties !== undefined && opts.properties.size > 0; + // Graph-degree cardinality predicates (t_9bee8f0b): a post-rank filter + // that can drop ranked rows, so it shares the frontmatter-filter + // overfetch. Absent / empty leaves the pool and results byte-identical. + const degreeFilters = opts.degreeFilters ?? []; + const hasDegreeFilter = degreeFilters.length > 0; // An explicit visibility scope can also drop ranked rows, so it // shares the property filter's overfetch. The default (no scope) // path does NOT overfetch up front - all-untagged vaults stay @@ -749,7 +755,8 @@ export async function search( // no ownership filtering, so untagged vaults stay byte-identical. const agentScope = normalizeAgentScope(opts.agentScope); const hasAgentScopeRequest = agentScope !== null; - const hasFrontmatterFilter = hasPropertyFilter || hasVisibilityRequest || hasAgentScopeRequest; + const hasFrontmatterFilter = + hasPropertyFilter || hasVisibilityRequest || hasAgentScopeRequest || hasDegreeFilter; // MMR and traversal both need a candidate pool wider than `limit`: // MMR diversifies from it, and traversal seeds expansion from it (a @@ -882,8 +889,14 @@ export async function search( const propFiltered = hasPropertyFilter ? applyPropertyFilter(statusFiltered, opts.properties!, config.vault, frontmatterCache) : statusFiltered; + // Graph-degree cardinality predicates (t_9bee8f0b): drop rows whose + // backlink/outlink counts fail the predicates, backed by the + // link-graph degree index. No-op when no predicate was requested. + const degreeFiltered = hasDegreeFilter + ? applyDegreeFilter(propFiltered, degreeFilters, store) + : propFiltered; const visible = applyVisibilityScope( - propFiltered, + degreeFiltered, visibilityScope, config.vault, frontmatterCache, @@ -894,7 +907,7 @@ export async function search( agentScope !== null ? applyAgentScope(visible, agentScope, config.vault, frontmatterCache) : visible; - return { preVisibility: propFiltered.length, visible: scoped, capHit }; + return { preVisibility: degreeFiltered.length, visible: scoped, capHit }; }; let assembled = assemble(rankLimit); diff --git a/src/core/search/types.ts b/src/core/search/types.ts index b3a39523..21ded395 100644 --- a/src/core/search/types.ts +++ b/src/core/search/types.ts @@ -5,6 +5,7 @@ */ import type { VaultIgnoreRule } from "../vault-scope/defaults.ts"; +import type { DegreePredicate } from "./property-filter.ts"; export type { VaultIgnoreRule }; @@ -550,6 +551,14 @@ export interface SearchOptions { * filter (existing behaviour). */ readonly properties?: ReadonlyMap>; + /** + * Graph-degree cardinality predicates (t_9bee8f0b). Each predicate + * selects notes by backlink/outlink count (orphans `backlinks = 0`, + * hubs `outlinks >= N`, etc.); predicates are ANDed and applied as a + * post-rank phase backed by the link-graph degree index. Absent / + * empty = no filter, byte-identical to prior behaviour. + */ + readonly degreeFilters?: ReadonlyArray; /** * Per-query MMR override (v0.13.0). Absent uses the resolved config * default; `1` disables diversification for this query. diff --git a/src/mcp/search-tools.ts b/src/mcp/search-tools.ts index e7573b2a..71508f07 100644 --- a/src/mcp/search-tools.ts +++ b/src/mcp/search-tools.ts @@ -25,6 +25,7 @@ import { } from "../core/search/index.ts"; import { normalizeSessionFocus, parseStructuredRecallQueryDocument } from "../core/search/index.ts"; import type { BrainSearchResult, SearchOutcome } from "../core/search/index.ts"; +import { parseDegreePredicate, type DegreePredicate } from "../core/search/property-filter.ts"; import { searchAcrossVaults } from "../core/search/cross-vault.ts"; import { RECALL_PROFILE_NAMES } from "../core/search/profiles.ts"; import { fileContextRecall } from "../core/brain/file-recall.ts"; @@ -151,6 +152,12 @@ const SEARCH_INPUT_SCHEMA: Record = { items: { type: "string" }, }, }, + degree: { + type: "array", + description: + "Graph-degree predicates over backlink/outlink counts, e.g. 'backlinks=0' (orphans) or 'outlinks>=5' (hubs); ANDed. Absent = no filter.", + items: { type: "string" }, + }, visibility: { type: "array", description: @@ -379,6 +386,31 @@ function parseVisibilityArgument(raw: unknown): string[] | undefined { return out; } +/** + * Validate the `degree` argument (array of `` strings) + * into a predicate list. A malformed shape or predicate throws + * INVALID_PARAMS rather than silently dropping the filter. + */ +function parseDegreeArgument(raw: unknown): DegreePredicate[] | undefined { + if (raw === undefined || raw === null) return undefined; + if (!Array.isArray(raw)) { + throw new MCPError(INVALID_PARAMS, "argument 'degree' must be an array of strings"); + } + const out: DegreePredicate[] = []; + for (const item of raw) { + if (typeof item !== "string") { + throw new MCPError(INVALID_PARAMS, "argument 'degree' must contain only strings"); + } + try { + out.push(parseDegreePredicate(item)); + } catch (e) { + if (e instanceof SearchError) throw searchErrorToMcp(e); + throw e; + } + } + return out; +} + function parseReinforceArgument(raw: unknown): string[] | undefined { if (raw === undefined || raw === null) return undefined; if (!Array.isArray(raw)) { @@ -494,6 +526,7 @@ async function toolBrainSearch( : undefined; const focusSession = coerceStringOptional(args, "focus_session", 128); const properties = parsePropertiesArgument(args["properties"]); + const degreeFilters = parseDegreeArgument(args["degree"]); const visibility = parseVisibilityArgument(args["visibility"]); const agentScope = coerceStringOptional(args, "agent_scope", 128); const reinforce = parseReinforceArgument(args["reinforce"]); @@ -527,6 +560,7 @@ async function toolBrainSearch( ...(profile !== undefined ? { profile } : {}), ...(disclosure === "cards" ? { disclosure: "cards" as const } : {}), ...(properties !== undefined ? { properties } : {}), + ...(degreeFilters !== undefined ? { degreeFilters } : {}), ...(visibility !== undefined ? { visibility } : {}), ...(agentScope !== undefined ? { agentScope } : {}), ...(structuredQuery !== undefined ? { structuredQuery } : {}), diff --git a/tests/core/brain/graph-index.test.ts b/tests/core/brain/graph-index.test.ts index 5595636b..4fa7d9b0 100644 --- a/tests/core/brain/graph-index.test.ts +++ b/tests/core/brain/graph-index.test.ts @@ -14,7 +14,11 @@ import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { getGraphSnapshot, graphStats } from "../../../src/core/brain/link-graph/graph-index.ts"; +import { + degreeForPath, + getGraphSnapshot, + graphStats, +} from "../../../src/core/brain/link-graph/graph-index.ts"; import { indexVault } from "../../../src/core/search/indexer.ts"; import { Store } from "../../../src/core/search/store.ts"; import { makeConfig } from "../../helpers/search-fixtures.ts"; @@ -100,6 +104,33 @@ describe("getGraphSnapshot", () => { }); }); +describe("directed degree (backlinks / outlinks)", () => { + test("counts distinct out-links and back-links per note", async () => { + // hub -> a, hub -> b, a -> b. So: + // hub: outlinks 2, backlinks 0 + // a: outlinks 1, backlinks 1 (from hub) + // b: outlinks 0, backlinks 2 (from hub and a) + // orphan: outlinks 0, backlinks 0 + writeFileSync(join(vault, "hub.md"), `# hub\n\n${link(["a", "b"])}\n`); + writeFileSync(join(vault, "a.md"), `# a\n\n${link(["b"])}\n`); + writeFileSync(join(vault, "b.md"), "# b\n\nNo outbound links.\n"); + writeFileSync(join(vault, "orphan.md"), "# orphan\n\nNothing.\n"); + await indexVault(config); + const store = await Store.open(config, { mode: "read" }); + try { + const snap = getGraphSnapshot(store); + expect(degreeForPath(snap, "hub.md")).toEqual({ backlinks: 0, outlinks: 2 }); + expect(degreeForPath(snap, "a.md")).toEqual({ backlinks: 1, outlinks: 1 }); + expect(degreeForPath(snap, "b.md")).toEqual({ backlinks: 2, outlinks: 0 }); + expect(degreeForPath(snap, "orphan.md")).toEqual({ backlinks: 0, outlinks: 0 }); + // Unknown path is treated as an orphan, not an error. + expect(degreeForPath(snap, "does-not-exist.md")).toEqual({ backlinks: 0, outlinks: 0 }); + } finally { + await store.close(); + } + }); +}); + describe("graphStats", () => { test("reports node/edge counts and top-degree nodes from the snapshot", async () => { writeClique(); diff --git a/tests/core/search/degree-filter-search.test.ts b/tests/core/search/degree-filter-search.test.ts new file mode 100644 index 00000000..65d13f36 --- /dev/null +++ b/tests/core/search/degree-filter-search.test.ts @@ -0,0 +1,66 @@ +/** + * Task Q3 (t_9bee8f0b): graph-degree cardinality predicates end-to-end + * through `search()`. + * + * Acceptance coverage: + * - filters select notes by backlink/outlink count, matching the graph + * index degree data (orphans `backlinks=0`, hubs `outlinks>=N`); + * - queries without degree predicates behave byte-identically. + */ + +import { test, expect, beforeEach, afterEach, describe } from "bun:test"; + +import { indexVault } from "../../../src/core/search/indexer.ts"; +import { search } from "../../../src/core/search/search.ts"; +import { parseDegreePredicate } from "../../../src/core/search/property-filter.ts"; +import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts"; + +describe("degree predicates in search", () => { + let vault: string; + let dbPath: string; + let cleanup: () => void; + + beforeEach(() => { + ({ vault, dbPath, cleanup } = createTempVault("degree")); + }); + afterEach(() => cleanup()); + + async function seed(): Promise> { + // hub -> a, hub -> b, a -> b. Shared token "lattice" so all match. + writeMd(vault, "hub.md", "# hub\n\nlattice [[a]] and [[b]]"); + writeMd(vault, "a.md", "# a\n\nlattice [[b]]"); + writeMd(vault, "b.md", "# b\n\nlattice terminal"); + writeMd(vault, "orphan.md", "# orphan\n\nlattice alone"); + const cfg = makeConfig({ vault, dbPath }); + await indexVault(cfg); + return cfg; + } + + test("orphans: backlinks=0 selects notes nobody links to", async () => { + const cfg = await seed(); + const out = await search(cfg, { + query: "lattice", + limit: 10, + degreeFilters: [parseDegreePredicate("backlinks=0")], + }); + // hub and orphan have zero back-links; a and b are linked to. + expect(out.results.map((r) => r.path).toSorted()).toEqual(["hub.md", "orphan.md"]); + }); + + test("hubs: outlinks>=2 selects notes with many out-links", async () => { + const cfg = await seed(); + const out = await search(cfg, { + query: "lattice", + limit: 10, + degreeFilters: [parseDegreePredicate("outlinks>=2")], + }); + expect(out.results.map((r) => r.path).toSorted()).toEqual(["hub.md"]); + }); + + test("no degree predicate is byte-identical to an unfiltered query", async () => { + const cfg = await seed(); + const baseline = await search(cfg, { query: "lattice", limit: 10 }); + const withEmpty = await search(cfg, { query: "lattice", limit: 10, degreeFilters: [] }); + expect(withEmpty.results.map((r) => r.path)).toEqual(baseline.results.map((r) => r.path)); + }); +}); diff --git a/tests/core/search/degree-filter.test.ts b/tests/core/search/degree-filter.test.ts new file mode 100644 index 00000000..25f9e560 --- /dev/null +++ b/tests/core/search/degree-filter.test.ts @@ -0,0 +1,115 @@ +/** + * Task Q3 (t_9bee8f0b): graph-degree cardinality predicates in the + * filter DSL. + * + * Acceptance coverage (parser + pure filter): + * - filters select notes by backlink/outlink count with all six + * comparison operators (orphans `= 0`, hubs `>= N`); + * - invalid predicate syntax rejects with a typed SearchError; + * - an empty predicate list is a byte-identical pass-through. + */ + +import { describe, expect, test } from "bun:test"; + +import { + applyDegreeFilters, + parseDegreePredicate, + type DegreePredicate, +} from "../../../src/core/search/property-filter.ts"; +import { SearchError } from "../../../src/core/search/types.ts"; + +describe("parseDegreePredicate", () => { + test("parses every operator and both fields", () => { + expect(parseDegreePredicate("backlinks=0")).toEqual({ + field: "backlinks", + op: "=", + value: 0, + }); + expect(parseDegreePredicate("outlinks>=5")).toEqual({ + field: "outlinks", + op: ">=", + value: 5, + }); + expect(parseDegreePredicate("backlinks!=2")).toEqual({ + field: "backlinks", + op: "!=", + value: 2, + }); + expect(parseDegreePredicate("outlinks<10")).toEqual({ field: "outlinks", op: "<", value: 10 }); + expect(parseDegreePredicate("backlinks<=3")).toEqual({ + field: "backlinks", + op: "<=", + value: 3, + }); + expect(parseDegreePredicate("outlinks>1")).toEqual({ field: "outlinks", op: ">", value: 1 }); + }); + + test("tolerates surrounding whitespace", () => { + expect(parseDegreePredicate(" backlinks >= 2 ")).toEqual({ + field: "backlinks", + op: ">=", + value: 2, + }); + }); + + test.each([ + "links=0", // unknown field + "backlinks", // no operator + "backlinks==2", // malformed operator + "outlinks>=-1", // negative value + "outlinks>=1.5", // non-integer + "backlinks>=abc", // non-numeric + "", // empty + ])("rejects %p with a typed SearchError", (expr) => { + let err: unknown; + try { + parseDegreePredicate(expr); + } catch (e) { + err = e; + } + expect(err).toBeInstanceOf(SearchError); + expect((err as SearchError).code).toBe("INVALID_INPUT"); + }); +}); + +describe("applyDegreeFilters", () => { + const rows = [{ path: "hub.md" }, { path: "a.md" }, { path: "b.md" }, { path: "orphan.md" }]; + const degrees: Record = { + "hub.md": { backlinks: 0, outlinks: 2 }, + "a.md": { backlinks: 1, outlinks: 1 }, + "b.md": { backlinks: 2, outlinks: 0 }, + "orphan.md": { backlinks: 0, outlinks: 0 }, + }; + const degreeOf = (path: string) => degrees[path] ?? { backlinks: 0, outlinks: 0 }; + + const run = (preds: DegreePredicate[]) => + applyDegreeFilters(rows, preds, degreeOf).map((r) => r.path); + + test("empty predicate list passes through byte-identically", () => { + const out = applyDegreeFilters(rows, [], degreeOf); + expect(out.map((r) => r.path)).toEqual(rows.map((r) => r.path)); + }); + + test("orphans: backlinks = 0", () => { + expect(run([parseDegreePredicate("backlinks=0")])).toEqual(["hub.md", "orphan.md"]); + }); + + test("hubs: outlinks >= 2", () => { + expect(run([parseDegreePredicate("outlinks>=2")])).toEqual(["hub.md"]); + }); + + test("!= operator", () => { + expect(run([parseDegreePredicate("backlinks!=0")])).toEqual(["a.md", "b.md"]); + }); + + test("< and <= operators", () => { + expect(run([parseDegreePredicate("outlinks<1")])).toEqual(["b.md", "orphan.md"]); + expect(run([parseDegreePredicate("backlinks<=1")])).toEqual(["hub.md", "a.md", "orphan.md"]); + }); + + test("multiple predicates are ANDed", () => { + expect(run([parseDegreePredicate("backlinks=0"), parseDegreePredicate("outlinks>=1")])).toEqual( + ["hub.md"], + ); + }); +}); From e16e25afc7220f6870fb380872469bf8ae3d6faa Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 22:17:35 +0000 Subject: [PATCH 11/14] feat(brain): guarded repair mode for doctor diagnostics Introduce the wave's diagnostics-signal model (issue class + detector + optional fixer + next-command hint) co-located with the doctor, and a guarded `o2b brain doctor --repair` mode built on it. - `--repair` previews planned fixes without writing (dry-run default); `--repair --apply` performs them and appends one typed `doctor-repair` event per fix. Re-running after apply is an idempotent no-op. - Fixers exist only for issue classes the doctor already detects: `wal-gap` closes a dangling dream workrun (an append-only write-ahead checkpoint log with no terminal phase) by appending the missing `interrupted` marker; `orphaned-reference` prunes a dead `evidenced_by` Brain link from a preference/retired record and its `## Origin` bullet. Broken structural links (supersedes, retired_by, superseded_by) are reported needs-review, never auto-removed. - Detected classes with no fixer are aggregated as unfixable, each with its next command from the signal registry (hints travel with the issue definition, never hardcoded in a formatter). - Plain doctor and `--strict` stay read-only and byte-identical; `--strict` cannot be combined with an applying repair. - MCP: optional `repair`/`apply` params on the existing brain_doctor tool (tool count stays 102). Co-Authored-By: Claude Fable 5 --- src/cli/brain/help-text.ts | 9 +- src/cli/brain/verbs/doctor.ts | 67 +++- src/core/brain/diagnostics.ts | 580 +++++++++++++++++++++++++++ src/core/brain/doctor.ts | 2 +- src/core/brain/types.ts | 12 + src/mcp/brain/health-tools.ts | 28 +- tests/cli/brain.test.ts | 41 ++ tests/core/brain.diagnostics.test.ts | 232 +++++++++++ tests/core/brain.types.test.ts | 2 + tests/mcp/brain.test.ts | 40 +- 10 files changed, 1006 insertions(+), 7 deletions(-) create mode 100644 src/core/brain/diagnostics.ts create mode 100644 tests/core/brain.diagnostics.test.ts diff --git a/src/cli/brain/help-text.ts b/src/cli/brain/help-text.ts index d4a76ad4..65d99eb2 100644 --- a/src/cli/brain/help-text.ts +++ b/src/cli/brain/help-text.ts @@ -41,7 +41,7 @@ Brain verbs (observing memory): snapshot diff Read-only diff between two snapshots, or snapshot vs live rollback Restore Brain/ from a snapshot (--list or ; --yes; --dry-run previews via the same diff renderer) - doctor Validate Brain invariants (--strict; --remediate [--dry-run]) + doctor Validate Brain invariants (--strict; --remediate/--repair [--apply]) hygiene Hygiene pipeline: scan findings; apply by ids (--dry-run) refresh Targeted recompile of stale derived pages (--stale [--dry-run]) anticipate Inspect or refresh the anticipatory context cache (--session) @@ -305,10 +305,13 @@ export const VERB_HELP: Record = { "(warm | stale | miss), or force a refresh first (TTL debounce applies).\n", doctor: "usage: o2b brain doctor [--vault ] [--json] [--strict]\n" + - " [--remediate [--dry-run]]\n" + + " [--remediate [--dry-run]] [--repair [--apply]]\n" + "Validate invariants. Warnings exit 0 (or 2 with --strict). Errors always exit 1.\n" + "--remediate builds a dependency-ordered repair plan and applies the\n" + - "auto-safe steps (content-hash re-stamp); --dry-run previews without writing.\n", + "auto-safe steps (content-hash re-stamp); --dry-run previews without writing.\n" + + "--repair previews safe fixes for detected classes (WAL gaps, orphaned\n" + + "references); --repair --apply performs them and logs one event per fix.\n" + + "Plain doctor and --strict stay read-only.\n", watchdog: "usage: o2b brain watchdog [--vault ] [--json] [--remediate [--dry-run]]\n" + " [--restore [--force-restore]] [--attempt ]\n" + diff --git a/src/cli/brain/verbs/doctor.ts b/src/cli/brain/verbs/doctor.ts index 672595b8..d6daccfb 100644 --- a/src/cli/brain/verbs/doctor.ts +++ b/src/cli/brain/verbs/doctor.ts @@ -1,5 +1,6 @@ import { resolveSearchConfig } from "../../../core/search/index.ts"; import { runDoctor } from "../../../core/brain/doctor.ts"; +import { applyRepair, type RepairOutcome } from "../../../core/brain/diagnostics.ts"; import { applyRemediation, collectDriftedSlugs, @@ -7,7 +8,7 @@ import { planRemediation, } from "../../../core/brain/health/remediation.ts"; import { loadBrainConfigDetailed, resolveHealth } from "../../../core/brain/policy.ts"; -import { brainVerbContext, fail, ok, parse } from "../helpers.ts"; +import { brainVerbContext, fail, ok, parse, resolveBrainAgent, usageError } from "../helpers.ts"; export async function cmdBrainDoctor(argv: string[]): Promise { const { flags } = parse(argv, { @@ -15,10 +16,34 @@ export async function cmdBrainDoctor(argv: string[]): Promise { strict: { type: "boolean" }, json: { type: "boolean" }, remediate: { type: "boolean" }, + repair: { type: "boolean" }, + apply: { type: "boolean" }, + agent: { type: "string" }, "dry-run": { type: "boolean" }, }); const { config, vault } = brainVerbContext(flags); + // Guarded repair mode (O2). Opt-in and dry-run by default; `--apply` + // performs the fixes. `--strict` stays read-only, so it can never be + // combined with an applying repair. Plain / `--strict` doctor below is + // untouched and byte-identical when `--repair` is absent. + if (flags["repair"]) { + if (flags["strict"] && flags["apply"]) { + return usageError("cannot combine --strict (read-only) with --repair --apply"); + } + const dryRun = !flags["apply"]; + try { + const outcome = applyRepair(vault, { + dryRun, + agent: resolveBrainAgent(flags, config), + configPath: config, + }); + return renderRepair(outcome, Boolean(flags["json"])); + } catch (exc) { + return fail(`repair failed: ${(exc as Error).message ?? exc}`); + } + } + let result; try { result = runDoctor(vault, { @@ -112,3 +137,43 @@ function runRemediate( ); return 0; } + +/** + * Render the guarded repair outcome. Dry-run lists what `--apply` would + * do; apply lists what was done. Needs-review instances and unfixable + * classes always print with their next-command hint (supplied by the + * diagnostics-signal definition, never hardcoded here). + */ +function renderRepair(outcome: RepairOutcome, json: boolean): number { + if (json) { + process.stdout.write(JSON.stringify(outcome, null, 2) + "\n"); + return 0; + } + + const verb = outcome.dryRun ? "would fix" : "fixed"; + for (const f of outcome.applied) { + process.stdout.write(`[${verb}] ${f.code}: ${f.detail}\n`); + } + for (const r of outcome.needsReview) { + process.stdout.write( + `[needs-review] ${r.code}: ${r.detail}${r.reason ? ` (${r.reason})` : ""}\n`, + ); + } + for (const u of outcome.unfixable) { + process.stdout.write(`[not auto-repairable] ${u.code} x${u.count}: run \`${u.nextCommand}\`\n`); + } + if ( + outcome.applied.length === 0 && + outcome.needsReview.length === 0 && + outcome.unfixable.length === 0 + ) { + ok("brain doctor --repair: nothing to fix"); + return 0; + } + ok( + `repair ${outcome.dryRun ? "dry-run" : "complete"}: ` + + `${outcome.applied.length} ${verb}, ${outcome.needsReview.length} needs-review, ` + + `${outcome.unfixable.length} not auto-repairable`, + ); + return 0; +} diff --git a/src/core/brain/diagnostics.ts b/src/core/brain/diagnostics.ts new file mode 100644 index 00000000..57c3d98c --- /dev/null +++ b/src/core/brain/diagnostics.ts @@ -0,0 +1,580 @@ +/** + * Diagnostics-signal model + guarded repair driver (Source pipeline + * integrity suite, O2, t_bd6cc4cb). + * + * This module is the ONE home for the diagnostics-signal shape the wave + * introduced: an issue class carries its own next-command hint and, when + * a safe deterministic repair exists, its fixer. Hints travel WITH the + * issue definition (the {@link DIAGNOSTIC_SIGNALS} registry) so no + * downstream formatter - not the repair preview, not the O3 operator + * snapshot - hardcodes a command string. Detection stays in its existing + * home: `doctor.ts` produces the issue stream and this module keys off it, + * never re-implementing a lint. + * + * Repair contract: + * - `doctor.ts` (plain / `--strict`) is read-only and byte-identical; + * nothing here runs unless the operator opts in with `--repair`. + * - `planRepair` is a pure read: it previews what `--apply` would do. + * - `applyRepair({ dryRun: true })` is the preview surface (writes + * nothing); `{ dryRun: false }` performs the fixes and appends ONE + * typed `doctor-repair` event per applied fix. + * - Every fixer is safe, deterministic, and idempotent: a second apply + * finds nothing to do and writes nothing. A detected instance a fixer + * cannot safely repair is reported as needs-review, never silently + * dropped and never pretended-fixed. + * + * Fixers exist ONLY for issue classes the doctor already detects: + * - `wal-gap` closes a dangling dream workrun (an append-only, + * write-ahead-style checkpoint log that never reached a terminal + * phase) by appending the missing terminal `interrupted` marker. + * Additive: forensic content is preserved, the gap is closed. + * - `orphaned-reference` prunes a dead `evidenced_by` wikilink (a + * Brain-managed `pref-`/`ret-`/`sig-` target with no file) from a + * preference or retired record. The removed pointer is captured in + * the typed event, so the change is auditable and recoverable. + * Broken structural links (`supersedes`, `retired_by`, + * `superseded_by`) are reported needs-review: removing one would drop + * lifecycle provenance or break a required field. + */ + +import { appendFileSync, existsSync, readdirSync, readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { normalizeAgentArgument } from "../agent-identity.ts"; +import { resolveAgentName } from "../config.ts"; +import { vaultRelative } from "../path-safety.ts"; +import { parseFrontmatter, writeFrontmatterAtomic } from "../vault.ts"; + +import { collectAllBasenames, runDoctor } from "./doctor.ts"; +import { scanDanglingWorkruns, WORKRUN_PHASE } from "./dream-workrun.ts"; +import { appendLogEvent } from "./log.ts"; +import { brainDirs } from "./paths.ts"; +import { parsePreference, parseRetired } from "./preference.ts"; +import { acquireLockSync } from "./sync-lockfile.ts"; +import { isoSecond } from "./time.ts"; +import { BRAIN_LOG_EVENT_KIND } from "./types.ts"; +import { normaliseWikilinkTarget } from "./wikilink.ts"; + +// ----- Diagnostics-signal model -------------------------------------------- + +/** + * One diagnostics signal: an issue class plus the exact CLI command an + * operator runs next to act on it. `autoRepairable` is true only when a + * fixer in this module can safely, idempotently repair the class. + */ +export interface DiagnosticSignal { + /** Stable code: a doctor issue code, a fixer code, or an O3 source code. */ + readonly code: string; + /** Short human label for the issue class. */ + readonly issueClass: string; + /** Exact next command to run (a structural CLI string, never prose). */ + readonly nextCommand: string; + /** True iff a fixer in this module repairs the class. */ + readonly autoRepairable: boolean; +} + +/** Fixer codes (the two auto-repairable classes this release ships). */ +export const REPAIR_CODE = Object.freeze({ + walGap: "wal-gap", + orphanedReference: "orphaned-reference", +} as const); + +/** + * Registry: the single home for every issue class this wave surfaces and + * its next-command hint. Doctor codes, fixer codes, and the O3 snapshot + * source codes all resolve here so a hint is defined exactly once. + */ +export const DIAGNOSTIC_SIGNALS: ReadonlyMap = new Map( + ( + [ + // --- Auto-repairable fixer classes --- + { + code: REPAIR_CODE.walGap, + issueClass: "dangling dream workrun (WAL gap)", + nextCommand: "o2b brain doctor --repair --apply", + autoRepairable: true, + }, + { + code: REPAIR_CODE.orphanedReference, + issueClass: "orphaned evidence reference", + nextCommand: "o2b brain doctor --repair --apply", + autoRepairable: true, + }, + // --- Doctor issue classes the fixers back (kept for hint lookup) --- + { + code: "dangling-workrun", + issueClass: "dangling dream workrun (WAL gap)", + nextCommand: "o2b brain doctor --repair --apply", + autoRepairable: true, + }, + { + code: "broken-wikilink", + issueClass: "broken frontmatter reference", + nextCommand: "o2b brain doctor --repair --apply", + autoRepairable: true, + }, + // --- Detected-but-not-auto-repairable doctor classes --- + { + code: "config-missing", + issueClass: "missing Brain config", + nextCommand: "o2b brain init", + autoRepairable: false, + }, + { + code: "config-invalid", + issueClass: "invalid Brain config", + nextCommand: "o2b brain doctor", + autoRepairable: false, + }, + { + code: "schema-version-unknown", + issueClass: "unknown schema version", + nextCommand: "o2b brain upgrade --apply", + autoRepairable: false, + }, + { + code: "principle-corrupted", + issueClass: "corrupted preference principle", + nextCommand: "o2b brain upgrade --apply", + autoRepairable: false, + }, + { + code: "content-hash-drift", + issueClass: "content-hash drift", + nextCommand: "o2b brain doctor --remediate", + autoRepairable: false, + }, + { + code: "duplicate-preferences", + issueClass: "duplicate preferences", + nextCommand: "o2b brain merge ", + autoRepairable: false, + }, + { + code: "orphan-evidence", + issueClass: "orphaned apply-evidence artifact", + nextCommand: "o2b brain audit", + autoRepairable: false, + }, + { + code: "broken-backlinks", + issueClass: "broken Brain backlink", + nextCommand: "o2b brain backlinks", + autoRepairable: false, + }, + { + code: "sync-conflict-log", + issueClass: "sync-conflict log copy", + nextCommand: "o2b brain doctor", + autoRepairable: false, + }, + { + code: "contradictory-preferences", + issueClass: "contradictory preferences", + nextCommand: "o2b brain health", + autoRepairable: false, + }, + { + code: "stale-claim", + issueClass: "stale confirmed preference", + nextCommand: "o2b brain stale", + autoRepairable: false, + }, + // --- O3 operator-snapshot source classes --- + { + code: "doctor-errors", + issueClass: "doctor errors", + nextCommand: "o2b brain doctor", + autoRepairable: false, + }, + { + code: "doctor-warnings", + issueClass: "doctor warnings", + nextCommand: "o2b brain doctor", + autoRepairable: false, + }, + { + code: "semantic-health", + issueClass: "semantic-health findings", + nextCommand: "o2b brain health", + autoRepairable: false, + }, + { + code: "hygiene-findings", + issueClass: "hygiene findings", + nextCommand: "o2b brain hygiene scan", + autoRepairable: false, + }, + { + code: "stale-notes", + issueClass: "stale entries", + nextCommand: "o2b brain stale", + autoRepairable: false, + }, + { + code: "review-queue", + issueClass: "review candidates pending", + nextCommand: "o2b brain review", + autoRepairable: false, + }, + { + code: "state-file", + issueClass: "state-file health", + nextCommand: "o2b brain init", + autoRepairable: false, + }, + ] satisfies ReadonlyArray + ).map((s) => [s.code, Object.freeze(s)]), +); + +/** + * Resolve a signal by code. Unknown codes fall back to a generic + * doctor-run hint so a newly-added lint still renders a next command + * without a formatter having to invent one. + */ +export function resolveSignal(code: string): DiagnosticSignal { + const known = DIAGNOSTIC_SIGNALS.get(code); + if (known) return known; + return Object.freeze({ + code, + issueClass: code, + nextCommand: "o2b brain doctor", + autoRepairable: false, + }); +} + +// ----- Repair plan shapes --------------------------------------------------- + +/** One planned fix (applicable) or a detected-but-needs-review instance. */ +export interface RepairItem { + /** Fixer code ({@link REPAIR_CODE}). */ + readonly code: string; + /** Stable, vault-relative target identifier the fix acts on. */ + readonly target: string; + /** True when a fixer can safely apply this; false = needs-review. */ + readonly applicable: boolean; + /** One-line human description of the planned action. */ + readonly detail: string; + /** Why the instance is needs-review (present iff `applicable` is false). */ + readonly reason?: string; +} + +/** A detected issue class with no fixer, aggregated for the preview. */ +export interface UnfixableClass { + readonly code: string; + readonly issueClass: string; + readonly count: number; + readonly nextCommand: string; +} + +export interface RepairPlan { + /** Every fixer finding: applicable fixes plus needs-review instances. */ + readonly fixes: ReadonlyArray; + /** Detected classes no fixer addresses, each with its next command. */ + readonly unfixable: ReadonlyArray; +} + +// ----- Fixers --------------------------------------------------------------- + +/** + * A fixer owns one auto-repairable class. `coversDoctorCode` is the + * doctor issue code the fixer represents, so the planner can exclude that + * code from the needs-a-different-tool `unfixable` list without parsing + * doctor messages. + */ +interface Fixer { + readonly code: string; + readonly coversDoctorCode: string; + plan(vault: string): RepairItem[]; + /** Apply one applicable item. Returns null on an idempotent no-op. */ + apply(vault: string, item: RepairItem): AppliedFix | null; +} + +/** Separator between the parts of an `orphaned-reference` target id. */ +const TARGET_SEP = "::"; +/** Raw frontmatter key for the derived evidence list. */ +const EVIDENCED_BY_KEY = "_evidenced_by"; +/** Brain-managed id prefixes: only these are pruned as orphaned. */ +const BRAIN_ID_RE = /^(pref|ret|sig)-/; + +function isBrokenBrainRef(raw: string, known: ReadonlySet): string | null { + const target = normaliseWikilinkTarget(raw); + if (!target) return null; + if (!BRAIN_ID_RE.test(target)) return null; // external / non-Brain link: leave it + if (known.has(target)) return null; + return target; +} + +const walGapFixer: Fixer = { + code: REPAIR_CODE.walGap, + coversDoctorCode: "dangling-workrun", + plan(vault: string): RepairItem[] { + return scanDanglingWorkruns(vault).map((path) => { + const rel = vaultRelative(path, vault); + return { + code: REPAIR_CODE.walGap, + target: rel, + applicable: true, + detail: `close dangling workrun ${rel} with a terminal 'interrupted' marker`, + }; + }); + }, + apply(vault: string, item: RepairItem): AppliedFix | null { + const path = join(vault, item.target); + if (!existsSync(path)) return null; + // Re-check: only act while the run is still dangling (idempotent). + if (!scanDanglingWorkruns(vault).some((p) => vaultRelative(p, vault) === item.target)) { + return null; + } + const line = + JSON.stringify({ + phase: WORKRUN_PHASE.interrupted, + at: new Date().toISOString(), + reason: "closed by doctor --repair", + }) + "\n"; + const existing = readFileSync(path, "utf8"); + const prefix = existing.length > 0 && !existing.endsWith("\n") ? "\n" : ""; + appendFileSync(path, prefix + line, "utf8"); + return { + code: item.code, + target: item.target, + detail: item.detail, + }; + }, +}; + +const orphanedReferenceFixer: Fixer = { + code: REPAIR_CODE.orphanedReference, + coversDoctorCode: "broken-wikilink", + plan(vault: string): RepairItem[] { + const known = collectAllBasenames(vault); + const items: RepairItem[] = []; + const dirs = brainDirs(vault); + + walkBrainRecords(dirs.preferences, "pref-", (path) => { + const pref = parsePreference(path); + const rel = vaultRelative(path, vault); + for (const raw of pref.evidenced_by) { + const dead = isBrokenBrainRef(raw, known); + if (dead) items.push(evidencePrune(rel, dead)); + } + if (pref.supersedes) { + const dead = isBrokenBrainRef(pref.supersedes, known); + if (dead) items.push(structuralReview(rel, "supersedes", dead)); + } + }); + + walkBrainRecords(dirs.retired, "ret-", (path) => { + const ret = parseRetired(path); + const rel = vaultRelative(path, vault); + for (const raw of ret.evidenced_by) { + const dead = isBrokenBrainRef(raw, known); + if (dead) items.push(evidencePrune(rel, dead)); + } + const retiredBy = isBrokenBrainRef(ret.retired_by, known); + if (retiredBy) items.push(structuralReview(rel, "retired_by", retiredBy)); + if (ret.superseded_by) { + const dead = isBrokenBrainRef(ret.superseded_by, known); + if (dead) items.push(structuralReview(rel, "superseded_by", dead)); + } + }); + + return items; + }, + apply(vault: string, item: RepairItem): AppliedFix | null { + const [rel, field, dead] = item.target.split(TARGET_SEP); + if (rel === undefined || field !== EVIDENCED_BY_KEY || dead === undefined) return null; + const path = join(vault, rel); + if (!existsSync(path)) return null; + const known = collectAllBasenames(vault); + + let handle: ReturnType; + try { + handle = acquireLockSync(path); + } catch { + return null; // contended: leave for a later run + } + try { + const [meta, body] = parseFrontmatter(path); + const arr = meta[EVIDENCED_BY_KEY]; + if (!Array.isArray(arr)) return null; + const next = arr.filter((raw) => { + if (typeof raw !== "string") return true; + const broken = isBrokenBrainRef(raw, known); + return broken !== dead; // drop exactly the still-dead target + }); + if (next.length === arr.length) return null; // idempotent no-op + meta[EVIDENCED_BY_KEY] = next; + // Keep the human `## Origin` prose consistent: drop the matching + // `- [[dead]]` bullet so the same dead target cannot re-surface as a + // body-side broken-backlink after the frontmatter is pruned. + const nextBody = removeOriginBullet(body, dead); + writeFrontmatterAtomic(path, meta, nextBody, { overwrite: true }); + return { code: item.code, target: item.target, detail: item.detail }; + } finally { + handle.release(); + } + }, +}; + +function evidencePrune(rel: string, dead: string): RepairItem { + return { + code: REPAIR_CODE.orphanedReference, + target: [rel, EVIDENCED_BY_KEY, dead].join(TARGET_SEP), + applicable: true, + detail: `prune orphaned evidence [[${dead}]] from ${rel}`, + }; +} + +function structuralReview(rel: string, field: string, dead: string): RepairItem { + return { + code: REPAIR_CODE.orphanedReference, + target: [rel, field, dead].join(TARGET_SEP), + applicable: false, + detail: `${rel} has a broken '${field}' link [[${dead}]]`, + reason: + "removing a structural lifecycle link would drop provenance or break a required field; " + + "reconcile it manually", + }; +} + +/** + * Remove every `- [[...]]` bullet from a preference/retired body so + * the `## Origin` prose stops naming a target the frontmatter no longer + * references. Only bullet lines whose wikilink normalises to `dead` are + * dropped; all other body content is preserved verbatim. + */ +function removeOriginBullet(body: string, dead: string): string { + const lines = body.split("\n"); + const kept = lines.filter((line) => { + const m = /^\s*-\s+(\[\[.+?\]\])\s*$/.exec(line); + if (!m) return true; + return normaliseWikilinkTarget(m[1]!) !== dead; + }); + return kept.join("\n"); +} + +/** Iterate `*.md` records under `dir`, skipping unparseable files. */ +function walkBrainRecords(dir: string, prefix: string, cb: (path: string) => void): void { + if (!existsSync(dir)) return; + for (const name of readdirSync(dir)) { + if (!name.endsWith(".md") || !name.startsWith(prefix)) continue; + try { + cb(join(dir, name)); + } catch { + // schema error - surfaced by the doctor, not this fixer's concern + } + } +} + +const FIXERS: ReadonlyArray = Object.freeze([walGapFixer, orphanedReferenceFixer]); +const FIXER_BY_CODE: ReadonlyMap = new Map(FIXERS.map((f) => [f.code, f])); +const COVERED_DOCTOR_CODES: ReadonlySet = new Set(FIXERS.map((f) => f.coversDoctorCode)); + +// ----- Planner -------------------------------------------------------------- + +/** + * Preview what a repair would do. Pure read: runs the doctor to enumerate + * detected classes, gathers every fixer's findings, and aggregates the + * classes no fixer addresses (each with its next-command hint). + */ +export function planRepair(vault: string): RepairPlan { + const fixes: RepairItem[] = []; + for (const fixer of FIXERS) fixes.push(...fixer.plan(vault)); + + const doctor = runDoctor(vault); + const counts = new Map(); + for (const issue of [...doctor.errors, ...doctor.warnings]) { + if (COVERED_DOCTOR_CODES.has(issue.code)) continue; + counts.set(issue.code, (counts.get(issue.code) ?? 0) + 1); + } + const unfixable: UnfixableClass[] = [...counts.entries()] + .map(([code, count]) => { + const sig = resolveSignal(code); + return { + code, + issueClass: sig.issueClass, + count, + nextCommand: sig.nextCommand, + }; + }) + .toSorted((a, b) => a.code.localeCompare(b.code)); + + return Object.freeze({ fixes: Object.freeze(fixes), unfixable: Object.freeze(unfixable) }); +} + +// ----- Apply ---------------------------------------------------------------- + +export interface AppliedFix { + readonly code: string; + readonly target: string; + readonly detail: string; + /** Absolute path of the log file the typed event landed in (apply only). */ + readonly logPath?: string; +} + +export interface RepairOutcome { + readonly dryRun: boolean; + /** Fixes that were (or, under dry-run, would be) applied. */ + readonly applied: ReadonlyArray; + /** Detected-but-needs-review instances a fixer will not touch. */ + readonly needsReview: ReadonlyArray; + /** Detected classes no fixer addresses. */ + readonly unfixable: ReadonlyArray; +} + +export interface ApplyRepairOptions { + /** True previews without writing; false performs the fixes. */ + readonly dryRun: boolean; + /** Wall clock for the typed event timestamps. Defaults to `new Date()`. */ + readonly now?: Date; + /** Agent identity recorded on each event. Resolver default when blank. */ + readonly agent?: string; + /** Config path for the agent-name resolver. */ + readonly configPath?: string; +} + +/** + * Run the guarded repair. `dryRun: true` returns exactly what would be + * applied and writes nothing; `dryRun: false` performs each applicable + * fix and appends one typed `doctor-repair` event per fix that actually + * changed disk. Idempotent: a second non-dry-run call finds nothing to do. + */ +export function applyRepair(vault: string, opts: ApplyRepairOptions): RepairOutcome { + const plan = planRepair(vault); + const needsReview = plan.fixes.filter((f) => !f.applicable); + const applicable = plan.fixes.filter((f) => f.applicable); + + if (opts.dryRun) { + const applied = applicable.map((f) => ({ code: f.code, target: f.target, detail: f.detail })); + return Object.freeze({ + dryRun: true, + applied: Object.freeze(applied), + needsReview: Object.freeze(needsReview), + unfixable: plan.unfixable, + }); + } + + const agent = normalizeAgentArgument(opts.agent ?? null) ?? resolveAgentName(opts.configPath); + const timestamp = isoSecond(opts.now ?? new Date()); + const applied: AppliedFix[] = []; + for (const item of applicable) { + const fixer = FIXER_BY_CODE.get(item.code); + if (!fixer) continue; + const result = fixer.apply(vault, item); + if (!result) continue; // idempotent no-op: nothing changed, no event + const res = appendLogEvent(vault, { + timestamp, + eventType: BRAIN_LOG_EVENT_KIND.doctorRepair, + body: { code: result.code, target: result.target, detail: result.detail, agent }, + }); + applied.push({ ...result, logPath: res.logPath }); + } + + return Object.freeze({ + dryRun: false, + applied: Object.freeze(applied), + needsReview: Object.freeze(needsReview), + unfixable: plan.unfixable, + }); +} diff --git a/src/core/brain/doctor.ts b/src/core/brain/doctor.ts index f1d90e6f..ac9c0a88 100644 --- a/src/core/brain/doctor.ts +++ b/src/core/brain/doctor.ts @@ -1445,7 +1445,7 @@ function checkEntities(vault: string, issues: DoctorIssue[]): void { } } -function collectAllBasenames(vault: string): ReadonlySet { +export function collectAllBasenames(vault: string): ReadonlySet { const out = new Set(); const dirs = brainDirs(vault); for (const d of [ diff --git a/src/core/brain/types.ts b/src/core/brain/types.ts index 5a6b53f9..a923d4d5 100644 --- a/src/core/brain/types.ts +++ b/src/core/brain/types.ts @@ -392,6 +392,18 @@ export const BRAIN_LOG_EVENT_KIND = { * an unchanged vault promotes nothing and stays byte-identical. */ sourceCitation: "source-citation", + /** + * `doctor-repair` (Source pipeline integrity suite, O2, t_bd6cc4cb) - the + * guarded `o2b brain doctor --repair --apply` mode performed one targeted + * fix for an issue class the doctor already detects. One event per applied + * fix. Payload carries the diagnostics-signal `code` (e.g. `wal-gap`, + * `orphaned-reference`), the `target` (vault-relative path or field ref the + * fix touched), a one-line `detail`, and the `agent`. A dry-run preview + * writes nothing and emits no event; re-running after a successful apply is + * a no-op because the underlying issue is gone, so the timeline never + * double-counts a fix. + */ + doctorRepair: "doctor-repair", } as const; export type BrainLogEventKind = (typeof BRAIN_LOG_EVENT_KIND)[keyof typeof BRAIN_LOG_EVENT_KIND]; diff --git a/src/mcp/brain/health-tools.ts b/src/mcp/brain/health-tools.ts index 45814b2c..ee64efef 100644 --- a/src/mcp/brain/health-tools.ts +++ b/src/mcp/brain/health-tools.ts @@ -9,6 +9,7 @@ import { resolveSearchConfig } from "../../core/search/index.ts"; import { collectMaintenanceActions } from "../../core/brain/maintenance/collect.ts"; import { runDoctor } from "../../core/brain/doctor.ts"; +import { applyRepair } from "../../core/brain/diagnostics.ts"; import type { ServerContext, ToolDefinition } from "../tool-contract.ts"; import { coerceBool, coerceFormat } from "../coerce.ts"; import { vaultRelativeSafe } from "./shared.ts"; @@ -20,6 +21,21 @@ async function toolBrainDoctor( const strict = coerceBool(args, "strict"); const format = coerceFormat(args); + // Guarded repair mode (O2). Opt-in and dry-run by default; `apply` + // performs the fixes. `strict` stays read-only and cannot apply. + const repair = coerceBool(args, "repair"); + if (repair) { + const apply = coerceBool(args, "apply"); + if (strict && apply) { + throw new Error("brain_doctor: cannot combine strict (read-only) with repair + apply"); + } + const outcome = applyRepair(ctx.vault, { + dryRun: !apply, + ...(ctx.configPath !== null ? { configPath: ctx.configPath } : {}), + }); + return { format, repair: outcome }; + } + const result = runDoctor(ctx.vault, { strict, dbPath: resolveSearchConfig({ vault: ctx.vault, configPath: ctx.configPath ?? undefined }) @@ -118,7 +134,7 @@ export const HEALTH_TOOLS: ReadonlyArray = Object.freeze([ { name: "brain_doctor", description: - "Validate `Brain/` invariants: status-vs-folder consistency, frontmatter validity, duplicate ids, ISO parsing, log header parsing. Read-only.", + "Validate `Brain/` invariants (status-vs-folder, frontmatter, duplicate ids, ISO, log headers). Read-only by default; `repair` previews safe fixes for detected classes (WAL gaps, orphaned references), `repair`+`apply` performs them and logs one event per fix.", inputSchema: { type: "object", properties: { @@ -126,6 +142,16 @@ export const HEALTH_TOOLS: ReadonlyArray = Object.freeze([ type: "boolean", description: "When true, warnings demote `ok` to false (CLI exit-code parity).", }, + repair: { + type: "boolean", + description: + "Preview safe fixes for issue classes the doctor detects (dry-run). Read-only unless `apply` is also set.", + }, + apply: { + type: "boolean", + description: + "With `repair`, perform the fixes and log one typed event per fix; otherwise `repair` is a dry-run preview.", + }, format: { type: "string", enum: ["markdown", "json"], diff --git a/tests/cli/brain.test.ts b/tests/cli/brain.test.ts index f178a11d..dda67d21 100644 --- a/tests/cli/brain.test.ts +++ b/tests/cli/brain.test.ts @@ -1200,6 +1200,47 @@ describe("brain doctor", () => { // we accept either non-zero code as a regression guard. expect([1, 2]).toContain(r.returncode); }); + + test("--repair previews without writing; --repair --apply performs the fix", async () => { + await bootstrap(); + // Plant a dangling dream workrun (WAL gap): last phase is non-terminal. + const runs = join(vault, "Brain", "log", "dream-runs"); + mkdirSync(runs, { recursive: true }); + const wr = join(runs, "run-cli.jsonl"); + writeFileSync( + wr, + JSON.stringify({ phase: "started", at: "2026-07-18T00:00:00.000Z", run_id: "run-cli" }) + + "\n", + "utf8", + ); + const before = readFileSync(wr, "utf8"); + + // Dry-run preview: exit 0, previews the fix, writes nothing. + const preview = await runCli(["brain", "doctor", "--vault", vault, "--repair"], { + env: { OPEN_SECOND_BRAIN_CONFIG: config }, + }); + expect(preview.returncode).toBe(0); + expect(preview.stdout).toContain("would fix"); + expect(preview.stdout).toContain("wal-gap"); + expect(readFileSync(wr, "utf8")).toBe(before); + + // Apply: exit 0, performs the fix, closes the gap. + const applied = await runCli(["brain", "doctor", "--vault", vault, "--repair", "--apply"], { + env: { OPEN_SECOND_BRAIN_CONFIG: config }, + }); + expect(applied.returncode).toBe(0); + expect(applied.stdout).toContain("fixed"); + expect(readFileSync(wr, "utf8")).toContain("interrupted"); + }); + + test("--strict --repair --apply is refused (read-only guard)", async () => { + await bootstrap(); + const r = await runCli( + ["brain", "doctor", "--vault", vault, "--strict", "--repair", "--apply"], + { env: { OPEN_SECOND_BRAIN_CONFIG: config } }, + ); + expect(r.returncode).toBe(2); + }); }); // ── §9 scan-inline CLI ────────────────────────────────────────────────────── diff --git a/tests/core/brain.diagnostics.test.ts b/tests/core/brain.diagnostics.test.ts new file mode 100644 index 00000000..081f24a0 --- /dev/null +++ b/tests/core/brain.diagnostics.test.ts @@ -0,0 +1,232 @@ +/** + * Tests for src/core/brain/diagnostics.ts - the diagnostics-signal model + * and the guarded `doctor --repair` driver (O2, t_bd6cc4cb). + * + * The model registry, the two fixers (WAL-gap dangling-workrun close and + * orphaned evidenced_by prune), dry-run preview, apply + typed event, and + * idempotency each get their own case. Fixtures use the canonical writers + * so the inputs match what real Brain operations produce; the dangling + * workrun is written by hand because no public writer leaves one behind. + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { + applyRepair, + DIAGNOSTIC_SIGNALS, + planRepair, + REPAIR_CODE, + resolveSignal, +} from "../../src/core/brain/diagnostics.ts"; +import { brainConfigPath, brainDirs, dreamWorkrunPath } from "../../src/core/brain/paths.ts"; +import { DEFAULT_BRAIN_CONFIG_YAML } from "../../src/core/brain/policy.ts"; +import { writePreference } from "../../src/core/brain/preference.ts"; +import { readLogDay, listLogDates } from "../../src/core/brain/log-jsonl.ts"; +import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts"; + +let tmp: string; + +beforeEach(() => { + tmp = mkdtempSync(join(tmpdir(), "o2b-diagnostics-")); + const dirs = brainDirs(tmp); + for (const d of [ + dirs.brain, + dirs.inbox, + dirs.processed, + dirs.preferences, + dirs.retired, + dirs.log, + dirs.snapshots, + ]) { + mkdirSync(d, { recursive: true }); + } + atomicWriteFileSync(brainConfigPath(tmp), DEFAULT_BRAIN_CONFIG_YAML); +}); + +afterEach(() => { + rmSync(tmp, { recursive: true, force: true }); +}); + +/** Write a dangling workrun (last phase is neither finalized nor interrupted). */ +function writeDanglingWorkrun(runId: string): string { + const path = dreamWorkrunPath(tmp, runId); + mkdirSync(join(tmp, "Brain", "log", "dream-runs"), { recursive: true }); + writeFileSync( + path, + JSON.stringify({ phase: "started", at: "2026-07-18T00:00:00.000Z", run_id: runId }) + + "\n" + + JSON.stringify({ phase: "cluster_complete", at: "2026-07-18T00:00:01.000Z", run_id: runId }) + + "\n", + "utf8", + ); + return path; +} + +function countRepairEvents(): number { + let n = 0; + for (const date of listLogDates(tmp)) { + for (const e of readLogDay(tmp, date).entries) { + if (e.eventType === "doctor-repair") n += 1; + } + } + return n; +} + +describe("diagnostics-signal model", () => { + test("fixer codes are registered as auto-repairable with a next command", () => { + for (const code of [REPAIR_CODE.walGap, REPAIR_CODE.orphanedReference]) { + const sig = DIAGNOSTIC_SIGNALS.get(code); + expect(sig).toBeDefined(); + expect(sig!.autoRepairable).toBe(true); + expect(sig!.nextCommand.length).toBeGreaterThan(0); + expect(sig!.nextCommand.startsWith("o2b ")).toBe(true); + } + }); + + test("resolveSignal falls back to a generic doctor hint for unknown codes", () => { + const sig = resolveSignal("some-brand-new-lint"); + expect(sig.code).toBe("some-brand-new-lint"); + expect(sig.autoRepairable).toBe(false); + expect(sig.nextCommand).toBe("o2b brain doctor"); + }); + + test("O3 source classes carry their own next-command hint (not hardcoded downstream)", () => { + expect(resolveSignal("stale-notes").nextCommand).toBe("o2b brain stale"); + expect(resolveSignal("hygiene-findings").nextCommand).toBe("o2b brain hygiene scan"); + expect(resolveSignal("review-queue").nextCommand).toBe("o2b brain review"); + }); +}); + +describe("planRepair", () => { + test("a clean vault plans no fixes and lists no unfixable classes", () => { + const plan = planRepair(tmp); + expect(plan.fixes).toEqual([]); + expect(plan.unfixable).toEqual([]); + }); + + test("detects a dangling workrun as an applicable wal-gap fix", () => { + writeDanglingWorkrun("run-abc"); + const plan = planRepair(tmp); + const wal = plan.fixes.filter((f) => f.code === REPAIR_CODE.walGap); + expect(wal).toHaveLength(1); + expect(wal[0]!.applicable).toBe(true); + expect(wal[0]!.target).toContain("run-abc"); + }); + + test("detects a dead evidenced_by as an applicable orphaned-reference fix", () => { + writePreference(tmp, { + slug: "alpha", + topic: "alpha", + principle: "rule", + created_at: "2026-05-14T10:00:00Z", + unconfirmed_until: "2026-05-28T10:00:00Z", + status: "unconfirmed", + evidenced_by: ["[[sig-never-existed]]"], + }); + const plan = planRepair(tmp); + const orphan = plan.fixes.filter((f) => f.code === REPAIR_CODE.orphanedReference); + expect(orphan).toHaveLength(1); + expect(orphan[0]!.applicable).toBe(true); + expect(orphan[0]!.target).toContain("sig-never-existed"); + }); + + test("does not prune an evidenced_by link that points outside the Brain id space", () => { + writePreference(tmp, { + slug: "beta", + topic: "beta", + principle: "rule", + created_at: "2026-05-14T10:00:00Z", + unconfirmed_until: "2026-05-28T10:00:00Z", + status: "unconfirmed", + evidenced_by: ["[[src/some/code.ts]]"], + }); + const plan = planRepair(tmp); + expect(plan.fixes.filter((f) => f.code === REPAIR_CODE.orphanedReference)).toEqual([]); + }); + + test("aggregates detected classes with no fixer as unfixable, each with a hint", () => { + // A duplicate id is a doctor error with no fixer. + const dupA = join(brainDirs(tmp).preferences, "pref-dup-a.md"); + const dupB = join(brainDirs(tmp).preferences, "pref-dup-b.md"); + const fm = + "---\nkind: brain-preference\nid: pref-collide\ncreated_at: 2026-05-14T10:00:00Z\n" + + "unconfirmed_until: 2026-05-28T10:00:00Z\ntags: []\ntopic: t\nprinciple: p\n" + + "provenance: observed\n_status: unconfirmed\n---\nbody\n"; + writeFileSync(dupA, fm, "utf8"); + writeFileSync(dupB, fm, "utf8"); + const plan = planRepair(tmp); + const dup = plan.unfixable.find((u) => u.code === "duplicate-id"); + expect(dup).toBeDefined(); + expect(dup!.nextCommand.startsWith("o2b ")).toBe(true); + }); +}); + +describe("applyRepair - dry run", () => { + test("previews fixes and writes nothing (no event, files untouched)", () => { + const wrPath = writeDanglingWorkrun("run-dry"); + const before = readFileSync(wrPath, "utf8"); + const out = applyRepair(tmp, { dryRun: true, now: new Date("2026-07-18T12:00:00Z") }); + expect(out.dryRun).toBe(true); + expect(out.applied.length).toBe(1); + expect(readFileSync(wrPath, "utf8")).toBe(before); // byte-identical + expect(countRepairEvents()).toBe(0); + }); +}); + +describe("applyRepair - apply", () => { + test("closes a dangling workrun and logs one typed event", () => { + const wrPath = writeDanglingWorkrun("run-apply"); + const out = applyRepair(tmp, { dryRun: false, now: new Date("2026-07-18T12:00:00Z") }); + expect(out.dryRun).toBe(false); + expect(out.applied.some((f) => f.code === REPAIR_CODE.walGap)).toBe(true); + // Terminal marker now present -> no longer dangling. + expect(readFileSync(wrPath, "utf8")).toContain("interrupted"); + expect(countRepairEvents()).toBe(1); + // Idempotent: a second apply finds nothing and logs nothing new. + const again = applyRepair(tmp, { dryRun: false, now: new Date("2026-07-18T12:05:00Z") }); + expect(again.applied).toEqual([]); + expect(countRepairEvents()).toBe(1); + }); + + test("prunes a dead evidenced_by and logs one typed event per fix", () => { + writePreference(tmp, { + slug: "gamma", + topic: "gamma", + principle: "rule", + created_at: "2026-05-14T10:00:00Z", + unconfirmed_until: "2026-05-28T10:00:00Z", + status: "unconfirmed", + evidenced_by: ["[[sig-never-existed]]"], + }); + const out = applyRepair(tmp, { dryRun: false, now: new Date("2026-07-18T12:00:00Z") }); + expect(out.applied.some((f) => f.code === REPAIR_CODE.orphanedReference)).toBe(true); + expect(countRepairEvents()).toBe(1); + // The dead reference is gone from both the frontmatter and the body + // Origin prose; re-running is a no-op. + const prefFile = join(brainDirs(tmp).preferences, "pref-gamma.md"); + expect(readFileSync(prefFile, "utf8")).not.toContain("sig-never-existed"); + const again = applyRepair(tmp, { dryRun: false, now: new Date("2026-07-18T12:05:00Z") }); + expect(again.applied).toEqual([]); + expect(countRepairEvents()).toBe(1); + }); + + test("a broken structural retired_by link is reported needs-review, never applied", () => { + // A retired record whose retired_by points at a missing preference. + const retDir = brainDirs(tmp).retired; + const path = join(retDir, "ret-old.md"); + const fm = + "---\nkind: brain-retired\nid: ret-old\ncreated_at: 2026-05-14T10:00:00Z\n" + + "tags: []\ntopic: t\nprinciple: p\nprovenance: observed\n" + + "retired_at: 2026-06-01T10:00:00Z\nretired_by: '[[pref-vanished]]'\n" + + "retired_reason: superseded-by-context\n_status: retired\n_evidenced_by: []\n---\nbody\n"; + writeFileSync(path, fm, "utf8"); + const out = applyRepair(tmp, { dryRun: false }); + expect(out.needsReview.some((f) => f.target.includes("retired_by"))).toBe(true); + expect(out.applied).toEqual([]); + expect(existsSync(path)).toBe(true); + expect(readFileSync(path, "utf8")).toContain("pref-vanished"); // untouched + }); +}); diff --git a/tests/core/brain.types.test.ts b/tests/core/brain.types.test.ts index ff5cb2c2..3df99d46 100644 --- a/tests/core/brain.types.test.ts +++ b/tests/core/brain.types.test.ts @@ -116,6 +116,8 @@ describe("BRAIN_* const enums", () => { "tension", // Source pipeline integrity suite (t_a3d1adb0) inline citation promotion "source-citation", + // Source pipeline integrity suite (t_bd6cc4cb) guarded doctor repair + "doctor-repair", ]); const actual = new Set(Object.values(BRAIN_LOG_EVENT_KIND)); expect(actual).toEqual(expected); diff --git a/tests/mcp/brain.test.ts b/tests/mcp/brain.test.ts index fb4c40f1..11582108 100644 --- a/tests/mcp/brain.test.ts +++ b/tests/mcp/brain.test.ts @@ -13,7 +13,15 @@ */ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync } from "node:fs"; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + rmSync, + writeFileSync, +} from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -732,6 +740,36 @@ describe("brain_doctor", () => { // Empty vault → empty actions array, but the field is always // present so MCP clients can rely on the shape. }); + + test("repair:true previews a WAL-gap fix without writing; apply performs it", async () => { + // Plant a dangling dream workrun (WAL gap). + const runs = join(vault, "Brain", "log", "dream-runs"); + mkdirSync(runs, { recursive: true }); + const wr = join(runs, "run-mcp.jsonl"); + writeFileSync( + wr, + JSON.stringify({ phase: "started", at: "2026-07-18T00:00:00.000Z", run_id: "run-mcp" }) + + "\n", + "utf8", + ); + const before = readFileSync(wr, "utf8"); + + const server = makeServer(); + await initialize(server); + + const preview = await call(server, "brain_doctor", { repair: true }); + expect(preview.result.isError).toBe(false); + const p = preview.result.structuredContent; + expect(p.repair.dryRun).toBe(true); + expect(p.repair.applied.length).toBe(1); + expect(readFileSync(wr, "utf8")).toBe(before); // nothing written + + const applied = await call(server, "brain_doctor", { repair: true, apply: true }); + const a = applied.result.structuredContent; + expect(a.repair.dryRun).toBe(false); + expect(a.repair.applied.length).toBe(1); + expect(readFileSync(wr, "utf8")).toContain("interrupted"); + }); }); // --------------------------------------------------------------------------- From b775dc3b0f897bf20896811c4a5c0ad1c5c6a037 Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 22:31:11 +0000 Subject: [PATCH 12/14] feat(brain): unified operator status snapshot with next-command hints Add `o2b brain status` (and the `brain_status` MCP counterpart): one read-only verb composing the operator signal sources that previously required polling six commands - doctor, semantic health, hygiene, stale scan, review candidates, active profile, and state-file health - into a single readable snapshot. - Every problem line carries the exact next command to run, looked up from the O2 diagnostics-signal registry (resolveSignal), so hints travel with the issue definition and are never hardcoded in the formatter. - A healthy vault prints a compact all-clear; each source is wrapped fail-soft so one broken surface degrades its line, not the snapshot. - Reads only. - MCP tool count goes 102 -> 103; frozen parity/count tests updated deliberately (mcp.test, removed-tools, brain-tools-parity), and brain_status is added to the preview-budget exempt list (bounded snapshot). Also point the review-queue signal hint at `o2b brain dream --dry-run` (the real surface for review candidates). Co-Authored-By: Claude Fable 5 --- src/cli/brain.ts | 3 + src/cli/brain/help-text.ts | 7 + src/cli/brain/verbs/index.ts | 1 + src/cli/brain/verbs/status.ts | 35 +++ src/cli/command-manifest.ts | 1 + src/core/brain/diagnostics.ts | 2 +- src/core/brain/operator-snapshot.ts | 241 +++++++++++++++++++++ src/mcp/brain/health-tools.ts | 33 +++ src/mcp/registry-guard.ts | 1 + tests/cli/brain.test.ts | 43 ++++ tests/core/brain.diagnostics.test.ts | 2 +- tests/core/brain.operator-snapshot.test.ts | 100 +++++++++ tests/mcp/brain-tools-parity.test.ts | 1 + tests/mcp/brain.test.ts | 37 ++++ tests/mcp/mcp.test.ts | 6 +- tests/mcp/removed-tools.test.ts | 4 +- 16 files changed, 513 insertions(+), 4 deletions(-) create mode 100644 src/cli/brain/verbs/status.ts create mode 100644 src/core/brain/operator-snapshot.ts create mode 100644 tests/core/brain.operator-snapshot.test.ts diff --git a/src/cli/brain.ts b/src/cli/brain.ts index 8c57b7b4..498e51f8 100644 --- a/src/cli/brain.ts +++ b/src/cli/brain.ts @@ -38,6 +38,7 @@ import { cmdBrainAnticipate, cmdBrainWatchdog, cmdBrainHealth, + cmdBrainStatus, cmdBrainHistory, cmdBrainActivation, cmdBrainTruth, @@ -223,6 +224,8 @@ export async function handleBrainSubcommand(argv: ReadonlyArray): Promis return await cmdBrainWatchdog(rest); case "health": return await cmdBrainHealth(rest); + case "status": + return await cmdBrainStatus(rest); case "history": return await cmdBrainHistory(rest); case "activation": diff --git a/src/cli/brain/help-text.ts b/src/cli/brain/help-text.ts index 65d99eb2..d63cdaab 100644 --- a/src/cli/brain/help-text.ts +++ b/src/cli/brain/help-text.ts @@ -42,6 +42,7 @@ Brain verbs (observing memory): rollback Restore Brain/ from a snapshot (--list or ; --yes; --dry-run previews via the same diff renderer) doctor Validate Brain invariants (--strict; --remediate/--repair [--apply]) + status Unified operator status snapshot with next-command hints hygiene Hygiene pipeline: scan findings; apply by ids (--dry-run) refresh Targeted recompile of stale derived pages (--stale [--dry-run]) anticipate Inspect or refresh the anticipatory context cache (--session) @@ -322,6 +323,12 @@ export const VERB_HELP: Record = { "Semantic-health report: contradictory confirmed preferences, recurring\n" + "concepts with no dedicated preference, and confirmed preferences on stale\n" + "evidence, plus a clean/watch/investigate verdict. Read-only.\n", + status: + "usage: o2b brain status [--vault ] [--json]\n" + + "Unified operator status snapshot. Composes doctor, semantic health,\n" + + "hygiene, stale scan, review candidates, active profile, and state-file\n" + + "health into one readable view. Every problem line carries the exact next\n" + + "command to run; a healthy vault prints a compact all-clear. Read-only.\n", history: "usage: o2b brain history [--vault ] [--json]\n" + "Render a preference's edit-history timeline (one entry per content\n" + diff --git a/src/cli/brain/verbs/index.ts b/src/cli/brain/verbs/index.ts index 1c97531e..6668cf2b 100644 --- a/src/cli/brain/verbs/index.ts +++ b/src/cli/brain/verbs/index.ts @@ -22,6 +22,7 @@ export { cmdBrainRollback } from "./rollback.ts"; export { cmdBrainDoctor } from "./doctor.ts"; export { cmdBrainWatchdog } from "./watchdog.ts"; export { cmdBrainHealth } from "./health.ts"; +export { cmdBrainStatus } from "./status.ts"; export { cmdBrainHistory } from "./history.ts"; export { cmdBrainActivation } from "./activation.ts"; export { cmdBrainTruth } from "./truth.ts"; diff --git a/src/cli/brain/verbs/status.ts b/src/cli/brain/verbs/status.ts new file mode 100644 index 00000000..06a99165 --- /dev/null +++ b/src/cli/brain/verbs/status.ts @@ -0,0 +1,35 @@ +import { + buildOperatorSnapshot, + renderOperatorSnapshot, +} from "../../../core/brain/operator-snapshot.ts"; +import { brainVerbContext, fail, parse } from "../helpers.ts"; + +/** + * `o2b brain status` - unified operator status snapshot (O3). Composes + * doctor, semantic health, hygiene, stale scan, review candidates, active + * profile, and state-file health into one readable snapshot. Every problem + * line carries the exact next command to run (from the diagnostics-signal + * definition). A healthy vault prints a compact all-clear. Read-only. + */ +export async function cmdBrainStatus(argv: string[]): Promise { + const { flags } = parse(argv, { + vault: { type: "string" }, + json: { type: "boolean" }, + }); + const { config, vault } = brainVerbContext(flags); + + let snapshot; + try { + snapshot = await buildOperatorSnapshot(vault, { configPath: config }); + } catch (exc) { + return fail(`status failed: ${(exc as Error).message ?? exc}`); + } + + if (flags["json"]) { + process.stdout.write(JSON.stringify(snapshot, null, 2) + "\n"); + return 0; + } + + process.stdout.write(renderOperatorSnapshot(snapshot)); + return 0; +} diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.ts index 4806eb9c..d4713dab 100644 --- a/src/cli/command-manifest.ts +++ b/src/cli/command-manifest.ts @@ -131,6 +131,7 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({ command("export", "Export active preferences"), command("explorer", "Open or export Brain graph explorer"), command("doctor", "Check Brain invariants"), + command("status", "Unified operator status snapshot with next-command hints"), command("hygiene", "Hygiene pipeline: scan findings, apply remediation plan"), command("refresh", "Targeted recompile of stale derived pages"), command("anticipate", "Inspect or refresh the anticipatory context cache"), diff --git a/src/core/brain/diagnostics.ts b/src/core/brain/diagnostics.ts index 57c3d98c..3220ae7f 100644 --- a/src/core/brain/diagnostics.ts +++ b/src/core/brain/diagnostics.ts @@ -214,7 +214,7 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap = new Map { code: "review-queue", issueClass: "review candidates pending", - nextCommand: "o2b brain review", + nextCommand: "o2b brain dream --dry-run", autoRepairable: false, }, { diff --git a/src/core/brain/operator-snapshot.ts b/src/core/brain/operator-snapshot.ts new file mode 100644 index 00000000..951be3a6 --- /dev/null +++ b/src/core/brain/operator-snapshot.ts @@ -0,0 +1,241 @@ +/** + * Unified operator status snapshot (Source pipeline integrity suite, O3, + * t_9f9c5466). + * + * One read-only composition of the signal sources an operator otherwise + * has to poll one command at a time - doctor, semantic health, hygiene, + * stale scan, review candidates, active profile, and state-file health - + * into a single readable snapshot. Every problem line carries the exact + * next command to run, and that command is looked up from the O2 + * diagnostics-signal registry ({@link resolveSignal}), never hardcoded in + * the renderer: a hint has exactly one home, the issue definition. + * + * Reads only. Each source is wrapped fail-soft so one broken surface + * (an unreadable index, a malformed config) degrades that line rather + * than sinking the whole snapshot. + */ + +import { existsSync, readdirSync, statSync } from "node:fs"; + +import { resolveSearchConfig } from "../search/index.ts"; +import { runDoctor } from "./doctor.ts"; +import { resolveSignal } from "./diagnostics.ts"; +import { runHygieneScan } from "./hygiene/scan.ts"; +import { brainConfigPath, brainDirs } from "./paths.ts"; +import { loadTemporalConfigSafe } from "./policy.ts"; +import { listProfiles } from "./portability/profiles.ts"; +import { buildReviewCandidates } from "./review-candidates.ts"; +import { buildTimelineIndex } from "./temporal/build-index.ts"; +import { findStaleEntries } from "./temporal/stale-watch.ts"; + +/** One problem line: an issue class, its summary, and the next command. */ +export interface SnapshotProblem { + /** Diagnostics-signal code driving the next-command lookup. */ + readonly code: string; + /** Human label for the issue class (from the signal definition). */ + readonly label: string; + /** One-line summary (counts / verdict) of the finding. */ + readonly detail: string; + /** Exact next command to run (from the signal definition). */ + readonly nextCommand: string; +} + +export interface SnapshotCounts { + readonly preferences: number; + readonly retired: number; + readonly inbox: number; +} + +export interface OperatorSnapshot { + readonly counts: SnapshotCounts; + /** stale preferences + signals + log files. */ + readonly staleTotal: number; + /** would_create + would_promote + would_retire from the dry-run dream. */ + readonly reviewQueue: number; + /** Active profile name, or null when none is set. */ + readonly activeProfile: string | null; + /** Presence of the two on-disk state files that back the vault. */ + readonly stateFiles: { + readonly config: boolean; + readonly searchIndex: boolean; + }; + /** Semantic-health verdict (clean | watch | investigate). */ + readonly healthVerdict: string; + /** Every problem line, each carrying its next command. */ + readonly problems: ReadonlyArray; + /** True when there are no problem lines. */ + readonly healthy: boolean; +} + +export interface BuildOperatorSnapshotOptions { + /** Config path for the search-index and profile lookups. */ + readonly configPath?: string; + /** Wall clock for stale/review scans. Defaults to `new Date()`. */ + readonly now?: Date; +} + +/** + * Compose the operator snapshot. Async because the review-candidates + * source runs a dry-run dream (read-only). Never mutates the vault. + */ +export async function buildOperatorSnapshot( + vault: string, + opts: BuildOperatorSnapshotOptions = {}, +): Promise { + const now = opts.now ?? new Date(); + const problems: SnapshotProblem[] = []; + const problem = (code: string, detail: string): void => { + const sig = resolveSignal(code); + problems.push({ code, label: sig.issueClass, detail, nextCommand: sig.nextCommand }); + }; + + // --- State-file health --- + const configPresent = existsSync(brainConfigPath(vault)); + let searchIndexPresent = false; + try { + searchIndexPresent = existsSync( + resolveSearchConfig({ vault, configPath: opts.configPath ?? undefined }).dbPath, + ); + } catch { + searchIndexPresent = false; + } + if (!configPresent) { + problem("state-file", "Brain config `_brain.yaml` is missing"); + } + + // --- Doctor + semantic health --- + let healthVerdict = "clean"; + try { + const doctor = runDoctor(vault, { now }); + if (doctor.errors.length > 0) { + problem("doctor-errors", `${doctor.errors.length} invariant error(s)`); + } + if (doctor.warnings.length > 0) { + problem("doctor-warnings", `${doctor.warnings.length} warning(s)`); + } + const verdict = doctor.semantic_health?.verdict; + if (verdict !== undefined) healthVerdict = verdict; + if (verdict !== undefined && verdict !== "clean") { + problem("semantic-health", `semantic-health verdict: ${verdict}`); + } + } catch { + // doctor never throws in practice; keep the snapshot resilient anyway. + } + + // --- Hygiene --- + try { + const hy = runHygieneScan(vault, { now }); + if (hy.findings.length > 0) { + problem("hygiene-findings", `${hy.findings.length} hygiene finding(s)`); + } + } catch { + /* fail-soft */ + } + + // --- Stale scan --- + let staleTotal = 0; + try { + const cfg = loadTemporalConfigSafe(vault); + const index = buildTimelineIndex(vault, {}); + const stale = findStaleEntries(index, vault, cfg, { now }); + staleTotal = + stale.stalePreferences.length + stale.staleSignals.length + stale.staleLogFiles.length; + if (staleTotal > 0) { + problem("stale-notes", `${staleTotal} stale entr${staleTotal === 1 ? "y" : "ies"}`); + } + } catch { + /* fail-soft */ + } + + // --- Review candidates (dry-run dream) --- + let reviewQueue = 0; + try { + const review = await buildReviewCandidates(vault, { now }); + reviewQueue = + review.would_create.length + review.would_promote.length + review.would_retire.length; + if (reviewQueue > 0) { + problem("review-queue", `${reviewQueue} review candidate(s) pending`); + } + } catch { + /* fail-soft */ + } + + // --- Active profile (informational) --- + let activeProfile: string | null = null; + if (opts.configPath !== undefined) { + try { + activeProfile = listProfiles(opts.configPath).active; + } catch { + activeProfile = null; + } + } + + return Object.freeze({ + counts: Object.freeze({ + preferences: countMarkdown(brainDirs(vault).preferences), + retired: countMarkdown(brainDirs(vault).retired), + inbox: countMarkdown(brainDirs(vault).inbox), + }), + staleTotal, + reviewQueue, + activeProfile, + stateFiles: Object.freeze({ config: configPresent, searchIndex: searchIndexPresent }), + healthVerdict, + problems: Object.freeze(problems), + healthy: problems.length === 0, + }); +} + +/** + * Render the snapshot as compact operator-readable text. A healthy vault + * prints a single all-clear line plus the summary; otherwise each problem + * prints with its next command underneath. + */ +export function renderOperatorSnapshot(snap: OperatorSnapshot): string { + const out: string[] = []; + const header = snap.healthy + ? "Vault status: all clear" + : `Vault status: ${snap.problems.length} problem(s)`; + out.push(header); + out.push( + ` preferences: ${snap.counts.preferences} retired: ${snap.counts.retired}` + + ` inbox: ${snap.counts.inbox}`, + ); + out.push(` stale: ${snap.staleTotal} review queue: ${snap.reviewQueue}`); + out.push(` active profile: ${snap.activeProfile ?? "none"}`); + out.push( + ` state files: config ${snap.stateFiles.config ? "ok" : "MISSING"}, ` + + `search index ${snap.stateFiles.searchIndex ? "present" : "absent"}`, + ); + out.push(` health: ${snap.healthVerdict}`); + + if (!snap.healthy) { + out.push(""); + out.push("Problems:"); + for (const p of snap.problems) { + out.push(` [${p.code}] ${p.detail}`); + out.push(` -> next: ${p.nextCommand}`); + } + } + out.push(""); + return out.join("\n"); +} + +/** Count `.md` files directly under `dir` (non-recursive). Missing dir = 0. */ +function countMarkdown(dir: string): number { + if (!existsSync(dir)) return 0; + try { + let n = 0; + for (const name of readdirSync(dir)) { + if (!name.endsWith(".md")) continue; + try { + if (statSync(`${dir}/${name}`).isFile()) n += 1; + } catch { + /* race: skip */ + } + } + return n; + } catch { + return 0; + } +} diff --git a/src/mcp/brain/health-tools.ts b/src/mcp/brain/health-tools.ts index ee64efef..f2195c19 100644 --- a/src/mcp/brain/health-tools.ts +++ b/src/mcp/brain/health-tools.ts @@ -10,6 +10,7 @@ import { resolveSearchConfig } from "../../core/search/index.ts"; import { collectMaintenanceActions } from "../../core/brain/maintenance/collect.ts"; import { runDoctor } from "../../core/brain/doctor.ts"; import { applyRepair } from "../../core/brain/diagnostics.ts"; +import { buildOperatorSnapshot } from "../../core/brain/operator-snapshot.ts"; import type { ServerContext, ToolDefinition } from "../tool-contract.ts"; import { coerceBool, coerceFormat } from "../coerce.ts"; import { vaultRelativeSafe } from "./shared.ts"; @@ -128,6 +129,20 @@ async function toolBrainHealth( }; } +// ----- brain_status -------------------------------------------------------- + +async function toolBrainStatus( + ctx: ServerContext, + args: Record, +): Promise> { + const format = coerceFormat(args); + const snapshot = await buildOperatorSnapshot( + ctx.vault, + ctx.configPath !== null ? { configPath: ctx.configPath } : {}, + ); + return { format, ...snapshot }; +} + // ----- Serializers --------------------------------------------------------- export const HEALTH_TOOLS: ReadonlyArray = Object.freeze([ @@ -181,4 +196,22 @@ export const HEALTH_TOOLS: ReadonlyArray = Object.freeze([ }, handler: toolBrainHealth, }, + { + name: "brain_status", + description: + "Unified operator status snapshot: composes doctor, semantic health, hygiene, stale scan, review candidates, active profile, and state-file health. Every problem carries the exact next command to run; a healthy vault reports all-clear. Read-only.", + inputSchema: { + type: "object", + properties: { + format: { + type: "string", + enum: ["markdown", "json"], + description: + "Output format hint. Structured result is identical; caller decides rendering.", + }, + }, + additionalProperties: false, + }, + handler: toolBrainStatus, + }, ]); diff --git a/src/mcp/registry-guard.ts b/src/mcp/registry-guard.ts index 61b7f37d..4dac8dc4 100644 --- a/src/mcp/registry-guard.ts +++ b/src/mcp/registry-guard.ts @@ -108,6 +108,7 @@ export const PREVIEW_BUDGET_EXEMPT: Readonly> = Object.fr brain_artifact_get: "the preview-budget escape hatch; truncating it would defeat itself", brain_health: "fixed-shape counters", brain_doctor: "issue list bounded by vault invariants; CLI surface renders full detail", + brain_status: "bounded operator snapshot: fixed counts plus a short problem list with hints", brain_recall_gate: "single verdict object", vault_health: "fixed list of manifest checks", brain_watchdog: "bounded probe report", diff --git a/tests/cli/brain.test.ts b/tests/cli/brain.test.ts index dda67d21..ad85f339 100644 --- a/tests/cli/brain.test.ts +++ b/tests/cli/brain.test.ts @@ -1243,6 +1243,49 @@ describe("brain doctor", () => { }); }); +describe("brain status", () => { + test("clean vault prints a compact all-clear", async () => { + await bootstrap(); + const r = await runCli(["brain", "status", "--vault", vault], { + env: { OPEN_SECOND_BRAIN_CONFIG: config }, + }); + expect(r.returncode).toBe(0); + expect(r.stdout).toContain("all clear"); + }); + + test("problem lines carry a next-command hint", async () => { + await bootstrap(); + const runs = join(vault, "Brain", "log", "dream-runs"); + mkdirSync(runs, { recursive: true }); + writeFileSync( + join(runs, "run-cli-status.jsonl"), + JSON.stringify({ + phase: "started", + at: "2026-07-18T00:00:00.000Z", + run_id: "run-cli-status", + }) + "\n", + "utf8", + ); + const r = await runCli(["brain", "status", "--vault", vault], { + env: { OPEN_SECOND_BRAIN_CONFIG: config }, + }); + expect(r.returncode).toBe(0); + expect(r.stdout).toContain("Problems:"); + expect(r.stdout).toContain("-> next: o2b "); + }); + + test("--json emits the structured snapshot", async () => { + await bootstrap(); + const r = await runCli(["brain", "status", "--vault", vault, "--json"], { + env: { OPEN_SECOND_BRAIN_CONFIG: config }, + }); + expect(r.returncode).toBe(0); + const payload = JSON.parse(r.stdout); + expect(typeof payload.healthy).toBe("boolean"); + expect(Array.isArray(payload.problems)).toBe(true); + }); +}); + // ── §9 scan-inline CLI ────────────────────────────────────────────────────── describe("brain scan-inline", () => { diff --git a/tests/core/brain.diagnostics.test.ts b/tests/core/brain.diagnostics.test.ts index 081f24a0..f0661a80 100644 --- a/tests/core/brain.diagnostics.test.ts +++ b/tests/core/brain.diagnostics.test.ts @@ -96,7 +96,7 @@ describe("diagnostics-signal model", () => { test("O3 source classes carry their own next-command hint (not hardcoded downstream)", () => { expect(resolveSignal("stale-notes").nextCommand).toBe("o2b brain stale"); expect(resolveSignal("hygiene-findings").nextCommand).toBe("o2b brain hygiene scan"); - expect(resolveSignal("review-queue").nextCommand).toBe("o2b brain review"); + expect(resolveSignal("review-queue").nextCommand).toBe("o2b brain dream --dry-run"); }); }); diff --git a/tests/core/brain.operator-snapshot.test.ts b/tests/core/brain.operator-snapshot.test.ts new file mode 100644 index 00000000..a3d4721b --- /dev/null +++ b/tests/core/brain.operator-snapshot.test.ts @@ -0,0 +1,100 @@ +/** + * Tests for src/core/brain/operator-snapshot.ts - the unified operator + * status snapshot (O3, t_9f9c5466). + * + * A healthy vault snapshots to an all-clear; a vault with detected issues + * produces problem lines, each carrying the exact next command supplied by + * the O2 diagnostics-signal registry (never hardcoded in the renderer). + */ + +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { + buildOperatorSnapshot, + renderOperatorSnapshot, +} from "../../src/core/brain/operator-snapshot.ts"; +import { resolveSignal } from "../../src/core/brain/diagnostics.ts"; +import { brainConfigPath, brainDirs } from "../../src/core/brain/paths.ts"; +import { DEFAULT_BRAIN_CONFIG_YAML } from "../../src/core/brain/policy.ts"; +import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts"; + +let tmp: string; + +beforeEach(() => { + tmp = mkdtempSync(join(tmpdir(), "o2b-snapshot-")); + const dirs = brainDirs(tmp); + for (const d of [ + dirs.brain, + dirs.inbox, + dirs.processed, + dirs.preferences, + dirs.retired, + dirs.log, + dirs.snapshots, + ]) { + mkdirSync(d, { recursive: true }); + } + atomicWriteFileSync(brainConfigPath(tmp), DEFAULT_BRAIN_CONFIG_YAML); +}); + +afterEach(() => { + rmSync(tmp, { recursive: true, force: true }); +}); + +describe("buildOperatorSnapshot", () => { + test("a clean vault snapshots to all-clear with no problems", async () => { + const snap = await buildOperatorSnapshot(tmp, { now: new Date("2026-07-18T12:00:00Z") }); + expect(snap.healthy).toBe(true); + expect(snap.problems).toEqual([]); + expect(snap.stateFiles.config).toBe(true); + const text = renderOperatorSnapshot(snap); + expect(text).toContain("all clear"); + expect(text).not.toContain("Problems:"); + }); + + test("a dangling workrun surfaces a doctor-warning problem with a next command", async () => { + const runs = join(tmp, "Brain", "log", "dream-runs"); + mkdirSync(runs, { recursive: true }); + writeFileSync( + join(runs, "run-x.jsonl"), + JSON.stringify({ phase: "started", at: "2026-07-18T00:00:00.000Z", run_id: "run-x" }) + "\n", + "utf8", + ); + const snap = await buildOperatorSnapshot(tmp, { now: new Date("2026-07-18T12:00:00Z") }); + expect(snap.healthy).toBe(false); + const dw = snap.problems.find((p) => p.code === "doctor-warnings"); + expect(dw).toBeDefined(); + // The hint travels with the signal definition, not the formatter. + expect(dw!.nextCommand).toBe(resolveSignal("doctor-warnings").nextCommand); + const text = renderOperatorSnapshot(snap); + expect(text).toContain("Problems:"); + expect(text).toContain(`-> next: ${dw!.nextCommand}`); + }); + + test("a missing config surfaces a state-file problem", async () => { + rmSync(brainConfigPath(tmp), { force: true }); + const snap = await buildOperatorSnapshot(tmp, { now: new Date("2026-07-18T12:00:00Z") }); + expect(snap.stateFiles.config).toBe(false); + const sf = snap.problems.find((p) => p.code === "state-file"); + expect(sf).toBeDefined(); + expect(sf!.nextCommand).toBe("o2b brain init"); + }); + + test("every problem line's next command matches its signal definition", async () => { + const runs = join(tmp, "Brain", "log", "dream-runs"); + mkdirSync(runs, { recursive: true }); + writeFileSync( + join(runs, "run-y.jsonl"), + JSON.stringify({ phase: "started", at: "2026-07-18T00:00:00.000Z", run_id: "run-y" }) + "\n", + "utf8", + ); + const snap = await buildOperatorSnapshot(tmp, { now: new Date("2026-07-18T12:00:00Z") }); + for (const p of snap.problems) { + expect(p.nextCommand).toBe(resolveSignal(p.code).nextCommand); + expect(p.nextCommand.startsWith("o2b ")).toBe(true); + } + }); +}); diff --git a/tests/mcp/brain-tools-parity.test.ts b/tests/mcp/brain-tools-parity.test.ts index 8247660b..b6866384 100644 --- a/tests/mcp/brain-tools-parity.test.ts +++ b/tests/mcp/brain-tools-parity.test.ts @@ -117,6 +117,7 @@ const FROZEN_BRAIN_TOOL_NAMES = [ "brain_skill_proposals", "brain_sources", "brain_stale_scan", + "brain_status", "brain_switch_vault", "brain_tension", "brain_tiers", diff --git a/tests/mcp/brain.test.ts b/tests/mcp/brain.test.ts index 11582108..e93d7372 100644 --- a/tests/mcp/brain.test.ts +++ b/tests/mcp/brain.test.ts @@ -772,6 +772,43 @@ describe("brain_doctor", () => { }); }); +// --------------------------------------------------------------------------- +// brain_status +// --------------------------------------------------------------------------- + +describe("brain_status", () => { + test("clean vault reports healthy with no problems", async () => { + const server = makeServer(); + await initialize(server); + const r = await call(server, "brain_status", {}); + expect(r.result.isError).toBe(false); + const s = r.result.structuredContent; + expect(s.healthy).toBe(true); + expect(s.problems).toEqual([]); + }); + + test("a dangling workrun surfaces a problem carrying a next command", async () => { + const runs = join(vault, "Brain", "log", "dream-runs"); + mkdirSync(runs, { recursive: true }); + writeFileSync( + join(runs, "run-status.jsonl"), + JSON.stringify({ phase: "started", at: "2026-07-18T00:00:00.000Z", run_id: "run-status" }) + + "\n", + "utf8", + ); + const server = makeServer(); + await initialize(server); + const r = await call(server, "brain_status", {}); + const s = r.result.structuredContent; + expect(s.healthy).toBe(false); + expect(s.problems.length).toBeGreaterThan(0); + for (const p of s.problems) { + expect(typeof p.nextCommand).toBe("string"); + expect(p.nextCommand.startsWith("o2b ")).toBe(true); + } + }); +}); + // --------------------------------------------------------------------------- // Advertised-tool deprecation guard // --------------------------------------------------------------------------- diff --git a/tests/mcp/mcp.test.ts b/tests/mcp/mcp.test.ts index 42a4074e..ffe29eb8 100644 --- a/tests/mcp/mcp.test.ts +++ b/tests/mcp/mcp.test.ts @@ -201,6 +201,8 @@ describe("tool listing", () => { "brain_agent_diff", "brain_doctor", "brain_health", + // Unified operator status snapshot (source-pipeline-integrity O3). + "brain_status", "brain_hygiene", "brain_mcp_landscape", // Canonical entity registry read surface (Memory Integrity Suite). @@ -643,7 +645,9 @@ describe("stdio loop", () => { // belief-lifecycle-decision-memory t_ac03214d) = 101. // + brain_tension (persisted-contradiction lifecycle, // belief-lifecycle-decision-memory t_0e3f2bee) = 102. - expect(list.result.tools.length).toBe(102); + // + brain_status (unified operator status snapshot, + // source-pipeline-integrity O3 t_9f9c5466) = 103. + expect(list.result.tools.length).toBe(103); }); test("returns parse error for invalid JSON", async () => { diff --git a/tests/mcp/removed-tools.test.ts b/tests/mcp/removed-tools.test.ts index efbe1032..c79170fa 100644 --- a/tests/mcp/removed-tools.test.ts +++ b/tests/mcp/removed-tools.test.ts @@ -169,5 +169,7 @@ test("the shadow surface is gone: no hidden tools, removed names unlisted", asyn // belief-lifecycle-decision-memory t_ac03214d) = 101. // + brain_tension (persisted-contradiction lifecycle, // belief-lifecycle-decision-memory t_0e3f2bee) = 102. - expect(list.result.tools.length).toBe(102); + // + brain_status (unified operator status snapshot, + // source-pipeline-integrity O3 t_9f9c5466) = 103. + expect(list.result.tools.length).toBe(103); }); From 5c298e76b4ce19b7d120cfcb8379b260fd8fc93b Mon Sep 17 00:00:00 2001 From: Sol Aitken Date: Sat, 18 Jul 2026 23:05:10 +0000 Subject: [PATCH 13/14] docs: source pipeline integrity wave docs and version 1.34.0 Co-Authored-By: Claude Fable 5 --- .claude-plugin/plugin.json | 2 +- .codex-plugin/plugin.json | 2 +- CHANGELOG.md | 25 +++++++++++++++++++ README.md | 2 +- .../source-pipeline-integrity/design.md | 6 +++-- docs/cli-reference.md | 21 ++++++++++++++++ docs/mcp.md | 11 ++++++++ openclaw.plugin.json | 2 +- package.json | 2 +- plugin.yaml | 2 +- plugins/codex/.codex-plugin/plugin.json | 2 +- plugins/hermes/plugin.yaml | 2 +- pyproject.toml | 2 +- 13 files changed, 70 insertions(+), 11 deletions(-) diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 748a583e..e67129ea 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "open-second-brain", - "version": "1.33.0", + "version": "1.34.0", "description": "Plugin-first second brain package for AI agents and humans.", "author": { "name": "Open Second Brain contributors" diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index fe337624..4bd90de7 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "open-second-brain", - "version": "1.33.0", + "version": "1.34.0", "description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.", "skills": "./skills", "hooks": "./hooks/hooks.json", diff --git a/CHANGELOG.md b/CHANGELOG.md index e5995fe8..a4b10ddf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.34.0] - 2026-07-18 + +A source pipeline integrity and operator tooling wave: eleven units that make vault intake trustworthy end to end (scoped and gated discovery, reconciled dispatch, deterministic pre-extraction, citation provenance), extend the query surface (configurable FTS tokenizer, graph-degree predicates), and give the operator a repairing doctor plus one consolidated status snapshot. Two shared kernels carry the wave: a gitignore-semantics path-scope engine and a diagnostics-signal registry whose next-command hints travel with the issue definitions. Everything stays deterministic (the kernel calls no LLM) and every new surface is byte-identical when unused. + +### Added + +- Nested `.gitignore` support in the hygiene file scan through a new path-scope engine (`src/core/fs/ignore.ts`): root and nested ignore files compose with git semantics (a deeper file scopes its own subtree, a nearer `!` re-include wins, `.git/info/exclude` participates); repositories without ignore files scan exactly as before and malformed patterns warn explicitly. +- Source ingest scoping: `--src-subpath` and `--exclude` on `o2b brain batch-plan` (and `src_subpath` / `exclude` on `brain_ingest_batch_plan`) restrict a monorepo ingest to one subtree with gitignore-style excludes through the same shared engine; a subpath escaping the source root rejects with a typed error. +- The schema `extractable` flag is enforced during page discovery: with a non-empty allowlist, pages whose `schema_type` is not listed are skipped before extraction, reported with a reason on the plan (`skipped_non_extractable`) and logged; an empty allowlist keeps today's behavior byte for byte and untyped pages are never gated. +- Deterministic no-LLM code-structure pre-extraction (`o2b brain pre-extract`, `pre_extract` on `brain_ingest_source`): classes, functions, imports, and inheritance edges from TypeScript, JavaScript, and Python sources as JSON entity/edge seeds handed to the agent ingest step; unknown languages report `extracted: false` with a reason, never a fake empty success. +- Dispatched-vs-ingested reconciliation (`o2b brain batch-plan --reconcile`, `reconcile` on `brain_ingest_batch_plan`): diffs a plan's dispatched set against checkpoint completions and names every silently-lost source; a complete plan reports an explicit empty gap; read-only and idempotent, a report rather than a retry. +- Inline citation promotion (`o2b brain scan-citations`): structural `[Source: , YYYY-MM-DD]` markers in note prose become dated `source-citation` events on the temporal timeline, stamped at the citation date and deduplicated on normalized name plus date; malformed markers are reported and skipped (`--strict` exits nonzero). +- Configurable FTS tokenizer: `search_fts_diacritics` (`0 | 1 | 2`) and `search_fts_stemmer` (`none | porter`) assemble the FTS5 tokenizer clause from validated allowlists; invalid values reject with a typed error naming the allowed set, changing the config requires an explicit `o2b search reindex`, and the CJK trigram path is untouched. +- Graph-degree cardinality predicates in the search filter DSL: `--degree` on `o2b search` (and `degree` on `brain_search`) filters by backlink and outlink counts with `=`, `!=`, `>`, `>=`, `<`, `<=` (orphans `backlinks=0`, hubs `outlinks>=N`), backed by directed degree data on the existing link-graph index. +- Doctor repair mode (`o2b brain doctor --repair`, `repair` / `apply` on `brain_doctor`): previews targeted fixes for doctor-detected issue classes (dangling workrun checkpoints, dead evidence links) without writing; `--repair --apply` performs them idempotently and logs one typed `doctor-repair` event per fix; structural lifecycle links and append-only audit history are reported needs-review or unfixable rather than half-fixed; plain doctor and `--strict` stay read-only. +- Unified operator status snapshot (`o2b brain status`, MCP `brain_status`): doctor, semantic health, hygiene, stale scan, review queue depth, active profile, and state-file health in one readable view; every problem line carries the exact next command supplied by the shared diagnostics-signal registry; a healthy vault prints a compact all-clear. +- Clean exit on early-closed stdout pipes: `o2b ... | head` (and `vault-log`) exits 0 on EPIPE, while any other stdout error now fails loudly instead of being silently swallowed. + +### Changed + +- The MCP tool surface grows from 102 to 103 tools (`brain_status`); `brain_ingest_batch_plan`, `brain_ingest_source`, `brain_doctor`, and `brain_search` gain the optional params listed above and stay byte-identical when the params are omitted. +- The hygiene scan replaces its static skip-dir list with gitignore-aware composition via the shared path-scope engine. +- Doctor diagnostics flow through a shared diagnostics-signal registry (issue class, detector, optional fixer, next-command hint) consumed by both `doctor --repair` and `o2b brain status`. + ## [1.33.0] - 2026-07-18 A belief lifecycle and decision memory wave: eleven units that give memories a full lifecycle (tombstone, supersede, temporal replacement, tension triage) and make decisions first-class, auditable citizens of the brain (records, ratings, commitment tiers, change receipts, governed recall). Everything is deterministic (the kernel still calls no LLM), every mutation is logged or receipted, and vaults that do not opt in behave byte-identically. @@ -6532,6 +6556,7 @@ plugin config (vault field)`, and exits with a clear - Sandbox vault and plugin manifest fixtures for tests. - GitHub release workflow for tag-based and manually dispatched releases. +[1.34.0]: https://github.com/itechmeat/open-second-brain/compare/v1.33.0...v1.34.0 [1.33.0]: https://github.com/itechmeat/open-second-brain/compare/v1.32.0...v1.33.0 [1.32.0]: https://github.com/itechmeat/open-second-brain/compare/v1.31.0...v1.32.0 [1.31.0]: https://github.com/itechmeat/open-second-brain/compare/v1.30.1...v1.31.0 diff --git a/README.md b/README.md index 16a05e61..a072a52d 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Open Second Brain plugs into [Hermes Agent](https://github.com/NousResearch/herm ## What is new -Open Second Brain 1.33.0 is a belief lifecycle and decision memory release. Memories now have a full lifecycle: an idempotent tombstone and supersede operation spans every memory type, facts can be replaced atomically at one instant with gap-free half-open validity intervals, a persisted claim graph answers "what is true now, what used to be true, what replaced it, what contests it" in one query, and injection prefers the tip of a supersedes chain while recall points superseded hits at their replacement. Decisions become first-class: recorded with assumption and review date (which opens a review obligation), rated and compared, tiered by commitment (exploring, leaning, decided, locked), receipted on every change with before/after/evidence trails, and resurfaced verbatim into matching sessions under per-session caps and spacing. Detected contradictions persist as triageable tension objects that warn at injection time until resolved, and transcript turns keep their authored time all the way into search ranking and time-bounded queries. The kernel still calls no LLM. +Open Second Brain 1.34.0 is a source pipeline integrity and operator tooling release. Getting sources into the vault is now trustworthy end to end: ingest can be scoped to one monorepo subtree with gitignore-style excludes, the hygiene scan honors nested `.gitignore` files with git semantics, the schema `extractable` flag actually gates page discovery, a deterministic no-LLM pre-pass extracts code structure (classes, functions, imports, inheritance) as seeds before agent ingest, and every batch plan can be reconciled so a source an agent silently dropped is named instead of lost. Prose citations like `[Source: name, YYYY-MM-DD]` become dated provenance events on the temporal timeline, the FTS tokenizer becomes configurable for non-English vaults, and search gains graph-degree predicates for orphan and hub queries. On the operator side, `o2b brain doctor --repair` turns diagnostics into guarded one-command fixes, `o2b brain status` consolidates every health signal into one snapshot with an exact next command per problem, and piping any CLI into `head` finally exits clean. The kernel still calls no LLM. ## Why diff --git a/docs/brainstorm/source-pipeline-integrity/design.md b/docs/brainstorm/source-pipeline-integrity/design.md index cdce167b..3620b371 100644 --- a/docs/brainstorm/source-pipeline-integrity/design.md +++ b/docs/brainstorm/source-pipeline-integrity/design.md @@ -101,8 +101,10 @@ only P1 before P2 and O2 before O3; every prefix of the ship order leaves - **EPIPE handling is scoped to stdout write failures**: an early-closed stdout pipe exits 0; all other I/O errors keep failing loudly. - **MCP parity**: agent-relevant new surfaces (status snapshot, repair - preview, degree predicates, citation scan) get MCP counterparts; frozen - parity lists and tool-count tests update accordingly. + preview, degree predicates) get MCP counterparts; frozen parity lists and + tool-count tests update accordingly. The citation scan ships CLI-only, + following the existing precedent for operator-driven scan verbs; an MCP + counterpart is deferred until an agent workflow needs it. ## File changes diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 64a2f4b3..61e7e5b5 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -262,6 +262,27 @@ o2b brain session-grep gains --since