diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 748a583e..e67129ea 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "open-second-brain", - "version": "1.33.0", + "version": "1.34.0", "description": "Plugin-first second brain package for AI agents and humans.", "author": { "name": "Open Second Brain contributors" diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index fe337624..4bd90de7 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "open-second-brain", - "version": "1.33.0", + "version": "1.34.0", "description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.", "skills": "./skills", "hooks": "./hooks/hooks.json", diff --git a/CHANGELOG.md b/CHANGELOG.md index e5995fe8..a4b10ddf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.34.0] - 2026-07-18 + +A source pipeline integrity and operator tooling wave: eleven units that make vault intake trustworthy end to end (scoped and gated discovery, reconciled dispatch, deterministic pre-extraction, citation provenance), extend the query surface (configurable FTS tokenizer, graph-degree predicates), and give the operator a repairing doctor plus one consolidated status snapshot. Two shared kernels carry the wave: a gitignore-semantics path-scope engine and a diagnostics-signal registry whose next-command hints travel with the issue definitions. Everything stays deterministic (the kernel calls no LLM) and every new surface is byte-identical when unused. + +### Added + +- Nested `.gitignore` support in the hygiene file scan through a new path-scope engine (`src/core/fs/ignore.ts`): root and nested ignore files compose with git semantics (a deeper file scopes its own subtree, a nearer `!` re-include wins, `.git/info/exclude` participates); repositories without ignore files scan exactly as before and malformed patterns warn explicitly. +- Source ingest scoping: `--src-subpath` and `--exclude` on `o2b brain batch-plan` (and `src_subpath` / `exclude` on `brain_ingest_batch_plan`) restrict a monorepo ingest to one subtree with gitignore-style excludes through the same shared engine; a subpath escaping the source root rejects with a typed error. +- The schema `extractable` flag is enforced during page discovery: with a non-empty allowlist, pages whose `schema_type` is not listed are skipped before extraction, reported with a reason on the plan (`skipped_non_extractable`) and logged; an empty allowlist keeps today's behavior byte for byte and untyped pages are never gated. +- Deterministic no-LLM code-structure pre-extraction (`o2b brain pre-extract`, `pre_extract` on `brain_ingest_source`): classes, functions, imports, and inheritance edges from TypeScript, JavaScript, and Python sources as JSON entity/edge seeds handed to the agent ingest step; unknown languages report `extracted: false` with a reason, never a fake empty success. +- Dispatched-vs-ingested reconciliation (`o2b brain batch-plan --reconcile`, `reconcile` on `brain_ingest_batch_plan`): diffs a plan's dispatched set against checkpoint completions and names every silently-lost source; a complete plan reports an explicit empty gap; read-only and idempotent, a report rather than a retry. +- Inline citation promotion (`o2b brain scan-citations`): structural `[Source: , YYYY-MM-DD]` markers in note prose become dated `source-citation` events on the temporal timeline, stamped at the citation date and deduplicated on normalized name plus date; malformed markers are reported and skipped (`--strict` exits nonzero). +- Configurable FTS tokenizer: `search_fts_diacritics` (`0 | 1 | 2`) and `search_fts_stemmer` (`none | porter`) assemble the FTS5 tokenizer clause from validated allowlists; invalid values reject with a typed error naming the allowed set, changing the config requires an explicit `o2b search reindex`, and the CJK trigram path is untouched. +- Graph-degree cardinality predicates in the search filter DSL: `--degree` on `o2b search` (and `degree` on `brain_search`) filters by backlink and outlink counts with `=`, `!=`, `>`, `>=`, `<`, `<=` (orphans `backlinks=0`, hubs `outlinks>=N`), backed by directed degree data on the existing link-graph index. +- Doctor repair mode (`o2b brain doctor --repair`, `repair` / `apply` on `brain_doctor`): previews targeted fixes for doctor-detected issue classes (dangling workrun checkpoints, dead evidence links) without writing; `--repair --apply` performs them idempotently and logs one typed `doctor-repair` event per fix; structural lifecycle links and append-only audit history are reported needs-review or unfixable rather than half-fixed; plain doctor and `--strict` stay read-only. +- Unified operator status snapshot (`o2b brain status`, MCP `brain_status`): doctor, semantic health, hygiene, stale scan, review queue depth, active profile, and state-file health in one readable view; every problem line carries the exact next command supplied by the shared diagnostics-signal registry; a healthy vault prints a compact all-clear. +- Clean exit on early-closed stdout pipes: `o2b ... | head` (and `vault-log`) exits 0 on EPIPE, while any other stdout error now fails loudly instead of being silently swallowed. + +### Changed + +- The MCP tool surface grows from 102 to 103 tools (`brain_status`); `brain_ingest_batch_plan`, `brain_ingest_source`, `brain_doctor`, and `brain_search` gain the optional params listed above and stay byte-identical when the params are omitted. +- The hygiene scan replaces its static skip-dir list with gitignore-aware composition via the shared path-scope engine. +- Doctor diagnostics flow through a shared diagnostics-signal registry (issue class, detector, optional fixer, next-command hint) consumed by both `doctor --repair` and `o2b brain status`. + ## [1.33.0] - 2026-07-18 A belief lifecycle and decision memory wave: eleven units that give memories a full lifecycle (tombstone, supersede, temporal replacement, tension triage) and make decisions first-class, auditable citizens of the brain (records, ratings, commitment tiers, change receipts, governed recall). Everything is deterministic (the kernel still calls no LLM), every mutation is logged or receipted, and vaults that do not opt in behave byte-identically. @@ -6532,6 +6556,7 @@ plugin config (vault field)`, and exits with a clear - Sandbox vault and plugin manifest fixtures for tests. - GitHub release workflow for tag-based and manually dispatched releases. +[1.34.0]: https://github.com/itechmeat/open-second-brain/compare/v1.33.0...v1.34.0 [1.33.0]: https://github.com/itechmeat/open-second-brain/compare/v1.32.0...v1.33.0 [1.32.0]: https://github.com/itechmeat/open-second-brain/compare/v1.31.0...v1.32.0 [1.31.0]: https://github.com/itechmeat/open-second-brain/compare/v1.30.1...v1.31.0 diff --git a/README.md b/README.md index 16a05e61..a072a52d 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Open Second Brain plugs into [Hermes Agent](https://github.com/NousResearch/herm ## What is new -Open Second Brain 1.33.0 is a belief lifecycle and decision memory release. Memories now have a full lifecycle: an idempotent tombstone and supersede operation spans every memory type, facts can be replaced atomically at one instant with gap-free half-open validity intervals, a persisted claim graph answers "what is true now, what used to be true, what replaced it, what contests it" in one query, and injection prefers the tip of a supersedes chain while recall points superseded hits at their replacement. Decisions become first-class: recorded with assumption and review date (which opens a review obligation), rated and compared, tiered by commitment (exploring, leaning, decided, locked), receipted on every change with before/after/evidence trails, and resurfaced verbatim into matching sessions under per-session caps and spacing. Detected contradictions persist as triageable tension objects that warn at injection time until resolved, and transcript turns keep their authored time all the way into search ranking and time-bounded queries. The kernel still calls no LLM. +Open Second Brain 1.34.0 is a source pipeline integrity and operator tooling release. Getting sources into the vault is now trustworthy end to end: ingest can be scoped to one monorepo subtree with gitignore-style excludes, the hygiene scan honors nested `.gitignore` files with git semantics, the schema `extractable` flag actually gates page discovery, a deterministic no-LLM pre-pass extracts code structure (classes, functions, imports, inheritance) as seeds before agent ingest, and every batch plan can be reconciled so a source an agent silently dropped is named instead of lost. Prose citations like `[Source: name, YYYY-MM-DD]` become dated provenance events on the temporal timeline, the FTS tokenizer becomes configurable for non-English vaults, and search gains graph-degree predicates for orphan and hub queries. On the operator side, `o2b brain doctor --repair` turns diagnostics into guarded one-command fixes, `o2b brain status` consolidates every health signal into one snapshot with an exact next command per problem, and piping any CLI into `head` finally exits clean. The kernel still calls no LLM. ## Why diff --git a/docs/brainstorm/source-pipeline-integrity/cli-output/claude.md b/docs/brainstorm/source-pipeline-integrity/cli-output/claude.md new file mode 100644 index 00000000..9770b165 --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/cli-output/claude.md @@ -0,0 +1,38 @@ +### Variant 1: Two shared kernels, cluster-local units + +- **Approach**: Extract exactly two new shared abstractions, each riding in with the first unit that needs it: a gitignore-style path-scope engine (`src/core/fs/ignore.ts`: nested ignore-file composition, nearer-`!`-wins, `.git/info/exclude` layering) introduced by Unit 1 and consumed by Unit 2's `--src-subpath`/`--exclude`; and a diagnostics-signal model (issue class + detector + optional fixer + next-command hint) introduced by Unit 9 and consumed by Unit 10's snapshot. Everything else lands as local changes to its existing module, organized into four clusters: scope/gate (1→2→3), ingest integrity (5, 4), provenance/query (6, 7, 8), operator surface (11, 9→10). The dependency spine is short - only 1→2 and 9→10 are hard edges; ship order 11, 1, 2, 3, 5, 4, 6, 7, 8, 9, 10 keeps every prefix coherent. +- **Trade-offs**: + - Pro: shared logic gets exactly one home (ignore matching, issue/hint model) without any infrastructure-only commit - each abstraction ships inside a feature commit with its tests. + - Pro: maximal prefix-shippability; a stalled wave still leaves `main` releasable after any unit. + - Pro: byte-identical opt-out is easy - the ignore engine replaces the static skip-dir list only where Unit 1 wires it; nothing else changes uninvoked. + - Con: the ingest path (Units 2, 3, 5, 4) stays four separate touch points in `ingest.ts`/`batch-plan.ts` rather than one explicit pipeline; a future wave may still want that consolidation. + - Con: the diagnostics model's shape is decided by Unit 9's needs and may need a small extension when Unit 10 arrives. +- **Complexity**: medium +- **Risk**: low + +### Variant 2: Source-pipeline spine + +- **Approach**: Treat Units 1-5 as stages of one explicit discovery pipeline abstraction in `src/core/brain/ingest/`: scope (ignore engine + subpath/exclude) → gate (`extractable` check) → pre-extract (deterministic code-structure pass) → dispatch → reconcile, each stage a typed hook on a shared `SourcePipeline` context. Operator tooling (9, 10) similarly reads a shared health-signal registry, and hygiene scan (Unit 1) becomes another consumer of the same scope stage. Units 6, 7, 8, 11 remain peripheral standalone changes. +- **Trade-offs**: + - Pro: strongest conceptual match to the wave theme - one choke point for "what enters the vault and how," which future ingest features slot into. + - Pro: reconciliation (Unit 4) and gating (Unit 3) become trivially testable as pipeline stages rather than call-site patches. + - Con: the pipeline scaffold is effectively an infrastructure commit, which the repo's conventions discourage, and it forces refactoring `ingest.ts`/`batch-plan.ts` before any user-visible unit ships - a stalled wave leaves a half-migrated pipeline. + - Con: highest risk to the byte-identical opt-out constraint, since existing ingest flow is rerouted through new machinery even when no new flag is set. + - Con: hygiene scan (Unit 1) importing from `brain/ingest` inverts sensible layering; the ignore engine would need to live below both anyway, eroding the "one spine" elegance. +- **Complexity**: large +- **Risk**: high + +### Variant 3: Eleven islands, convention-only sharing + +- **Approach**: Every unit lands independently in its own existing module with no new shared homes: Unit 1 implements ignore parsing inside `hygiene/`, Unit 2 imports that parser directly (a hygiene→ingest reach-across), Unit 10 calls the existing verb functions (`brain_health`, `brain_doctor`, etc.) and formats their outputs itself, and Unit 9 pairs fixers with doctor checks ad hoc inside `doctor.ts`. Ordering is nearly free - only Unit 2's import of Unit 1's parser is a hard edge. +- **Trade-offs**: + - Pro: cheapest to execute and the easiest to parallelize across agents; every commit is small, atomic, and independently revertable. + - Pro: near-perfect prefix-shippability with zero scaffolding to leave half-finished. + - Con: the ignore engine living under `hygiene/` while `ingest/` imports it violates the one-directional-layering direction, or else gets duplicated - either way conflicting with "shared logic gets exactly one home." + - Con: Unit 10 re-deriving and re-formatting signals from six verbs duplicates presentation logic and drifts as those verbs evolve; hints get hardcoded per call site instead of traveling with the issue. + - Con: defers exactly the consolidation debt the post-v1.30.1 refactor just paid down. +- **Complexity**: small +- **Risk**: medium + +### Recommended: Variant 1 +**Rationale**: Variant 1 delivers the two abstractions this wave genuinely reuses (ignore composition, diagnostics signals with hints) in their correct one-home locations, while avoiding Variant 2's infrastructure-first commit and its threat to the byte-identical opt-out and atomic-feature-commit conventions. Unlike Variant 3, it keeps layering one-directional - the ignore engine sits below both hygiene and ingest, and hints travel with issue definitions instead of being duplicated in the snapshot. Its short dependency spine (1→2, 9→10) preserves the constraint that a partially completed wave still ships as a coherent prefix of v1.34.0. diff --git a/docs/brainstorm/source-pipeline-integrity/cli-output/prompt.md b/docs/brainstorm/source-pipeline-integrity/cli-output/prompt.md new file mode 100644 index 00000000..ef238027 --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/cli-output/prompt.md @@ -0,0 +1,88 @@ +You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation. + +# Task + +This is a release wave of eleven kanban tasks for Open Second Brain (OSB). The wave theme is source pipeline integrity and operator tooling: everything about getting sources into the vault correctly (scoped, gated, reconciled, deterministically pre-extracted, provenance-carrying), querying what is there, and keeping the vault healthy. Brainstorm the wave architecture as a whole: where shared abstractions live, what the dependency spine is, and how the units cluster. Do not judge whether individual units are worth doing; all eleven ship. + +## Unit 1 (t_9654de80) - Nested .gitignore in hygiene file scan +Teach the hygiene file scan to honor `.gitignore` (and a project-local ignore file) with git-like nested composition: an ignore file deeper in the tree applies to its own subtree, and a nearer `!` re-include wins, replacing the single hardcoded skip-dir list. Code today: `src/core/hygiene/scan-repo.ts:75` uses a static skip-dir array; no gitignore parsing exists anywhere in the scan. The nearer-`!`-wins composition rule is the deliberate part; compose with `.git/info/exclude` semantics. + +## Unit 2 (t_e82101a5) - `--src-subpath` and `--exclude` on source ingest +Add `--src-subpath` and `--exclude` to source ingest/sync so a monorepo can be ingested from one subdirectory, excluding sibling packages. `--exclude` should compose with ignore handling rather than duplicate it. Incremental sync exists via `o2b watch` (command-manifest.ts:259); no subdir scoping anywhere today. + +## Unit 3 (t_ed856388) - Honor the schema `extractable` flag during page discovery +OSB already stores an `extractable` flag on packs via schema mutation (schema-mutate.ts:54 `set_extractable`, handler :260-263) but nothing consults it during page discovery/extraction. Add discovery-time enforcement: pages in non-extractable packs are skipped up front. Behavior-only wiring of an existing flag. + +## Unit 4 (t_d067a153) - Reconcile dispatched-vs-ingested documents in batch ingest +Large-folder ingest plans bounded batches (`planBatches`, src/core/brain/ingest/batch-plan.ts:125) and folds per-source completion into a checkpoint (`recordCompleted`, checkpoint.ts:169) and content manifest. Nothing compares the plan's dispatched set against the files that actually came back. Add a `reconcilePlan(planId)` helper next to the checkpoint, invoked after a batch/plan drains, warning on silently-lost sources. + +## Unit 5 (t_ef786747) - Local no-LLM code-structure extractor as a pre-ingest pass +A deterministic, no-LLM code-structure extractor that parses code sources into classes/functions/imports/inheritance edges as JSON, run as a pre-pass before agent-driven ingest to cut extraction tokens and produce deterministic entity/edge seeds. Must stay pure-stdlib for the default path. Scope it as a lightweight fallback pre-pass, complementing (not replacing) the detect-only codegraph partner integration (src/core/partner/codegraph.ts). Ingest consumer: src/core/brain/ingest/ingest.ts. + +## Unit 6 (t_a3d1adb0) - Inline `[Source: , YYYY-MM-DD]` prose citations into the temporal timeline +Parse citation markers embedded in note prose and promote each into the temporal timeline as a dated provenance entry. Timeline builds only from structured logs today (temporal/build-index.ts walks Brain/log/*.jsonl); provenance lives in provenance/provenance.ts + portability/origins.ts. New parser feeding existing temporal + provenance sinks; handle malformed dates and dedup against already-logged source events. NOTE the OSB constraint: no hardcoded natural-language word lists; the `[Source: ...]` marker is a structural syntax, which is acceptable, but date parsing must be structural. + +## Unit 7 (t_618f7211) - Configurable FTS tokenizer language/diacritic rules +The FTS tokenizer is hardcoded (`tokenize='unicode61 remove_diacritics 2'` in search/schema.ts). Expose tokenizer/language (stemming + diacritic rules) as configuration so non-English vaults index on language-appropriate rules, leaving the default intact. CJK is handled out-of-band via a separate fts_content column + trigram prefilter; the config must compose with that path, not replace it. Reindex command already exists (`o2b search reindex`). Config must not require language word lists. + +## Unit 8 (t_9bee8f0b) - Graph-degree cardinality predicates in the search/filter DSL +Add a cardinality predicate to the user-facing search/filter DSL filtering notes by the COUNT of link-graph relations: backlinks/outlinks `= 0` (orphans/leaves), `>= N` (hubs). Degree is already computed internally (link-graph/graph-index.ts:33 `degree` map, moc-audit backlink buckets); the DSL (src/core/search/property-filter.ts) matches only frontmatter key/values today. CLI surface in src/cli/search.ts. + +## Unit 9 (t_bd6cc4cb) - `o2b brain doctor --repair` +Add a guarded repair mode to the existing read-only doctor (doctor.ts) that performs targeted fixes for the issue classes doctor already detects (orphaned references, WAL gaps). Opt-in, dry-run/preview by default, `--strict` read-only behavior preserved as default. + +## Unit 10 (t_9f9c5466) - Unified operator status snapshot +One readable CLI health snapshot combining counts, stale + orphaned pages, review queue depth, active profile, and state-file health, with an actionable next-command hint printed on each problem line. The raw signals are already computed across brain_health, brain_doctor, brain_hygiene, brain_stale_scan, brain_review_candidates, brain_brief operator view; ship the consolidation + per-line hints only. + +## Unit 11 (t_2ed754d1) - Early-closed stdout pipe as clean exit in CLIs +Make `o2b` and `vault-log` treat an early-closed stdout pipe (EPIPE after `| head`) as a normal exit 0 instead of a nonzero failure. Entry points are bash wrappers exec-ing `bun run src/cli/main.ts`; main exits via `main(...).then((code) => process.exit(code))` (src/cli/main.ts:933). No EPIPE/SIGPIPE handling exists. Small and self-contained. + +# Project context + +Open Second Brain: TypeScript on Bun, CLI (`o2b`) + MCP server over an Obsidian-compatible Markdown vault, bun:sqlite + sqlite-vec hybrid search. Deterministic kernel: the core never calls an LLM; agent-driven steps live outside the kernel. + +Recent commits: +77513f2b feat: belief lifecycle and decision memory (v1.33.0) (#141) +61e93d24 fix(config): derive vault store reference from a keyed installation secret (#140) +9a649dd6 feat: memory write-path integrity and store safety wave (v1.32.0) (#139) +f2a037eb feat: today operator surface - dashboard, open loops, marker write-back (v1.31.0) (#138) +13bde6c3 refactor: remove all import cycles, decompose search.ts (v1.30.1) (#137) +fd5661f9 feat: governance visibility - vitals scorecard + batch-inflation lint (v1.30.0) (#136) +a99b0e71 feat(brain): add o2b brain vitals scorecard + batch-concept-inflation lint (#135) +70fb36e1 feat: operability, safety & first-run experience (v1.29.0) (#134) +ac26a675 feat: retrieval & ranking quality (v1.28.0) (#133) +5cd52e70 fix(hermes): resolve o2b when memory provider PATH is tiny (v1.27.1) (#131) + +Related files: +src/core/hygiene/scan-repo.ts, src/core/brain/ingest/{ingest.ts,batch-plan.ts,checkpoint.ts,content-manifest.ts}, src/core/schema/schema-mutate.ts, src/core/temporal/build-index.ts, src/core/provenance/provenance.ts, src/core/search/{schema.ts,property-filter.ts,fts.ts}, src/core/brain/link-graph/graph-index.ts, src/cli/main.ts, src/cli/search.ts, src/cli/brain/verbs/*, src/mcp/*, src/core/partner/codegraph.ts, scripts/o2b, scripts/vault-log + +Conventions: +- Each unit lands as one atomic conventional feature commit with its tests; infrastructure-only commits are discouraged. +- Post-v1.30.1 direction: shared choke points, one-directional layering, no import cycles; shared logic gets exactly one home. +- SOLID, KISS, DRY; constants extracted; typed errors surfaced explicitly; no silent do-nothing fallbacks; no stubs. +- Language-agnostic: no built-in natural-language word lists anywhere; structural signals and config-supplied vocabularies only. +- New CLI verbs get an MCP counterpart when they are agent-relevant; MCP tool descriptions are capped and parity-tested. +- Byte-identical opt-out: with new features unconfigured, existing outputs must not change. + +Constraints: +- Bun runtime, bun:sqlite; no new external dependencies for the default path (pure stdlib for parsing/extraction). +- Do not change existing public APIs; new params optional. +- The deterministic kernel calls no LLM. +- The wave ships as one PR / one release (v1.34.0); a partially completed wave must still ship coherently as a prefix. + +# Required output format + +Produce exactly 3 distinct architectural variants. For each variant: + +### Variant N: +- **Approach**: 2-3 sentences describing the variant. +- **Trade-offs**: bullet list of pros and cons. +- **Complexity**: small | medium | large +- **Risk**: low | medium | high + +After the three variants, add exactly one recommendation: + +### Recommended: Variant N +**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above. + +Output nothing outside of these sections. diff --git a/docs/brainstorm/source-pipeline-integrity/design.md b/docs/brainstorm/source-pipeline-integrity/design.md new file mode 100644 index 00000000..3620b371 --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/design.md @@ -0,0 +1,137 @@ +# Source pipeline integrity and operator tooling - one wave, eleven units + +**Status:** accepted +**Author:** Claude orchestrator (via feature-release-playbook) +**Audience:** implementation + +## Problem statement + +Sources enter the vault through paths that cannot be scoped or gated, batches +can silently lose documents, prose citations never reach the temporal layer, +and vault health signals are scattered across six read-only surfaces with no +repair action. This wave makes intake trustworthy end to end: scoped and +gated discovery, reconciled dispatch, deterministic pre-extraction, citation +provenance, configurable indexing, degree-aware querying, and an operator +toolchain that can both summarize and repair. + +## Scope + +Eleven kanban tasks ship as one PR and one release (v1.34.0): + +- `t_2ed754d1` early-closed stdout pipe exits clean in `o2b` / `vault-log` +- `t_9654de80` nested `.gitignore` composition in the hygiene file scan +- `t_e82101a5` `--src-subpath` / `--exclude` scoping on source ingest +- `t_ed856388` `extractable` flag honored during page discovery +- `t_ef786747` local no-LLM code-structure extractor as a pre-ingest pass +- `t_d067a153` dispatched-vs-ingested reconciliation for batch ingest plans +- `t_a3d1adb0` inline `[Source: , YYYY-MM-DD]` citations become dated + provenance events on the temporal timeline +- `t_618f7211` configurable FTS tokenizer language and diacritic rules +- `t_9bee8f0b` graph-degree cardinality predicates in the search/filter DSL +- `t_bd6cc4cb` `o2b brain doctor --repair` for the issue classes doctor + already detects +- `t_9f9c5466` unified operator status snapshot with per-problem + next-command hints + +## Out of scope + +- Rebuilding the reindex command (exists as `o2b search reindex`). +- Replacing or extending the codegraph partner integration; the pre-ingest + extractor is a fallback pre-pass, not a codegraph substitute. +- Tree-sitter or any non-stdlib parsing backend for the extractor. +- An interactive profile explorer or onboarding scaffolder (different theme). +- Repair actions for issue classes doctor does not already detect. +- `t_71f96533` (created_at bounds on raw-session grep) is excluded as a + duplicate: v1.33.0 already shipped `since` / `before` on session grep. + +## Chosen approach + +Consultant Variant 1: two shared kernels, cluster-local units. Exactly two +new shared abstractions, each riding inside the first feature commit that +needs it: + +1. **Path-scope engine** `src/core/fs/ignore.ts` - nested ignore-file + composition with git semantics (deeper file scopes its subtree, + nearer `!` re-include wins, `.git/info/exclude` layering). Introduced by + the hygiene-scan unit (P1), consumed by ingest scoping (P2). It sits + below both `hygiene/` and `brain/ingest/`, keeping layering + one-directional. +2. **Diagnostics-signal model** - issue class + detector + optional fixer + + next-command hint, introduced by `doctor --repair` (O2) and consumed by + the unified status snapshot (O3), so hints travel with issue definitions + instead of being duplicated in the snapshot formatter. + +Every other unit is a local change to its existing module. Four clusters: +scope/gate (P1 -> P2 -> P3), ingest integrity (P4, P5), provenance/query +(Q1, Q2, Q3), operator surface (O1, O2 -> O3). Hard dependency edges are +only P1 before P2 and O2 before O3; every prefix of the ship order leaves +`main` releasable. + +## Design decisions + +- **Ignore engine location** `src/core/fs/ignore.ts`, not under `hygiene/` + or `ingest/`: both consume it; one home, no reach-across imports. +- **Byte-identical opt-out everywhere**: unset config, absent flags, and + unconfigured features must leave existing outputs unchanged; + regression-tested per unit. +- **Explicit errors, no silent fallbacks**: malformed ignore patterns, + invalid tokenizer configs, invalid degree predicates, malformed repair + targets, and unknown citation dates surface as typed errors or explicit + warnings; nothing is silently skipped without a report. +- **Reconciliation is a report, not a retry**: `reconcilePlan(planId)` diffs + the dispatched set against completed entries and returns/warns the gap; + it does not re-dispatch (that stays operator-driven). +- **Extractor stays deterministic and stdlib-only**: structural parsing per + language family via regex/line grammar, emitting JSON entity/edge seeds + that ingest passes to the agent as pre-extracted facts; no natural-language + word lists, no LLM in the kernel. +- **Citation syntax is structural**: `[Source: , YYYY-MM-DD]` is a + fixed marker grammar with an ISO-shaped date; no language-specific parsing. + Dedup key is (normalized name, date) against already-logged source events. +- **Tokenizer config composes with CJK**: config selects FTS5 tokenizer + options (stemming/diacritics); the existing trigram prefilter path is + untouched. Changing the config requires an explicit `o2b search reindex`; + the CLI says so rather than reindexing implicitly. +- **Degree predicates read the existing graph index**: the DSL gains count + predicates over backlinks/outlinks backed by `graph-index.ts` degree data; + no new graph computation. +- **Repair is opt-in and previewed**: `doctor --repair` defaults to dry-run + preview; `--apply` performs fixes; `--strict` read-only behavior is + unchanged. Every applied fix logs a typed event. +- **EPIPE handling is scoped to stdout write failures**: an early-closed + stdout pipe exits 0; all other I/O errors keep failing loudly. +- **MCP parity**: agent-relevant new surfaces (status snapshot, repair + preview, degree predicates) get MCP counterparts; frozen parity lists and + tool-count tests update accordingly. The citation scan ships CLI-only, + following the existing precedent for operator-driven scan verbs; an MCP + counterpart is deferred until an agent workflow needs it. + +## File changes + +New: `src/core/fs/ignore.ts`, `src/core/brain/ingest/pre-extract.ts` (code +structure extractor), `src/core/brain/ingest/reconcile.ts`, +`src/core/temporal/citations.ts`, diagnostics-signal module next to +`doctor.ts`, snapshot verb module, tests per unit. + +Modified: `src/core/hygiene/scan-repo.ts`, ingest CLI/MCP surface and +`ingest.ts` / `batch-plan.ts` / `checkpoint.ts`, discovery path consuming +`extractable`, `src/core/search/schema.ts` + config for tokenizer, +`src/core/search/property-filter.ts` + `src/cli/search.ts` for degree +predicates, `src/cli/main.ts` / `scripts/*` for EPIPE, doctor CLI/MCP, +docs and manifests (version 1.34.0). + +## Risks and open questions + +- Gitignore composition has sharp edge cases (anchoring, directory-only + patterns, `!` precedence); mitigate with a property/table test suite + mirroring git's documented rules. Full `git check-ignore` parity is not + claimed; the supported subset is documented. +- The extractor's language coverage is bounded; unknown languages must fall + through to today's behavior explicitly (reported as unextracted, never a + fake empty result). +- Repair fixers touch vault files; every fixer needs a preview diff and an + idempotency test. +- The diagnostics-signal model shape is set by O2 and may need a small + extension when O3 lands; acceptable churn inside one branch. +- MCP tool-count and description-budget guards will need deliberate updates; + keep additions within the 300-char description cap. diff --git a/docs/brainstorm/source-pipeline-integrity/plan.md b/docs/brainstorm/source-pipeline-integrity/plan.md new file mode 100644 index 00000000..e61e2fa3 --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/plan.md @@ -0,0 +1,146 @@ +# Source pipeline integrity and operator tooling - implementation plan + +Ordering follows consultant Variant 1: two shared kernels ride inside the +first feature commit that needs them (P1 owns `src/core/fs/ignore.ts`, O2 +owns the diagnostics-signal model); every other unit is a local change to +its existing module. Hard dependency edges are only P1 before P2 and O2 +before O3; every prefix of the sequence leaves the branch releasable. +Each unit lands as one atomic conventional commit with its tests, formatted +(oxfmt) and lint-clean (oxlint, 134-warning/0-error baseline). + +Sequence: O1 -> P1 -> P2 -> P3 -> P4 -> P5 -> Q1 -> Q2 -> Q3 -> O2 -> O3 -> L + +## Tasks + +### Task O1 - clean exit on early-closed stdout pipe (`t_2ed754d1`, p3, standalone) +- **Files**: `src/cli/main.ts` (EPIPE-aware exit path), shared stdout-write + guard if needed by `vault-log`, tests. +- **Acceptance**: `o2b | head -1` exits 0 with no error + output for both `o2b` and `vault-log` entry points; EPIPE on stdout maps + to exit 0; any other I/O error still fails nonzero with its message; a + regression test simulates a closed-pipe write. +- **Depends on**: none. + +### Task P1 - nested gitignore composition in hygiene scan (`t_9654de80`, p3, path-scope anchor) +- **Files**: new `src/core/fs/ignore.ts` (ignore-rule engine: pattern parse, + nested composition, nearer-`!`-wins, `.git/info/exclude` layering), + `src/core/hygiene/scan-repo.ts` wiring, tests (table suite mirroring git + documented semantics). +- **Acceptance**: the hygiene scan skips paths ignored by root and nested + `.gitignore` files; a deeper ignore file scopes only its subtree; a nearer + `!` re-include wins over an outer ignore; `.git/info/exclude` participates; + repos without ignore files scan byte-identically to today (static skip-dir + behavior preserved as baseline); malformed patterns surface as explicit + warnings, never silent skips. +- **Depends on**: none. + +### Task P2 - source ingest scoping flags (`t_e82101a5`, p3) +- **Files**: ingest CLI/MCP surface (`--src-subpath`, `--exclude`), + `src/core/brain/ingest/*` walk scoping via `src/core/fs/ignore.ts` + pattern matching, tests. +- **Acceptance**: ingest of a monorepo scoped with `--src-subpath pkg/a` + processes only that subtree; `--exclude` patterns compose with ignore + handling (same engine, no duplicate matcher); a subpath outside the source + root rejects with a typed error; without the new flags ingest planning is + byte-identical to today. +- **Depends on**: P1 (`src/core/fs/ignore.ts`). + +### Task P3 - extractable flag gate in discovery (`t_ed856388`, p3) +- **Files**: page-discovery path (locate the pack iteration), gate consuming + the existing `extractable` schema flag, tests. +- **Acceptance**: pages in packs marked non-extractable are skipped before + extraction and reported as skipped-with-reason in the discovery result; + packs without the flag behave exactly as today; the skip is logged, not + silent; no schema mutation surface changes. +- **Depends on**: none. + +### Task P4 - no-LLM code-structure pre-ingest extractor (`t_ef786747`, p3) +- **Files**: new `src/core/brain/ingest/pre-extract.ts` (deterministic + stdlib-only structural extractor: classes/functions/imports/inheritance + edges as JSON seeds), wiring in `src/core/brain/ingest/ingest.ts` to pass + seeds to the agent step, CLI/MCP exposure of the pre-pass, tests. +- **Acceptance**: running the pre-pass on a TypeScript/JavaScript and Python + source produces deterministic JSON entity/edge seeds (same input, same + output); unknown languages are reported as unextracted, never a fake empty + success; ingest with the pre-pass off is byte-identical to today; no + natural-language word lists; no new dependencies. +- **Depends on**: none (P2's scoping optional at call site). + +### Task P5 - dispatched-vs-ingested reconciliation (`t_d067a153`, p3) +- **Files**: new `src/core/brain/ingest/reconcile.ts` (`reconcilePlan` + keyed on `planId` diffing `BatchPlan` dispatched set vs checkpoint + completed set), invocation after a batch/plan drains, CLI/MCP surfacing of + the gap report, tests. +- **Acceptance**: a plan whose agent response omits dispatched files yields + a reconciliation report naming each lost source; a fully completed plan + reports an empty gap; the report is a warning surface, not a retry; the + reconcile is idempotent and read-only over checkpoint state. +- **Depends on**: none. + +### Task Q1 - inline citation promotion to temporal timeline (`t_a3d1adb0`, p3) +- **Files**: new `src/core/temporal/citations.ts` (structural + `[Source: , YYYY-MM-DD]` marker parser + promotion into temporal + log/provenance sinks), scan surface (CLI/MCP), tests. +- **Acceptance**: a note containing a well-formed citation marker produces a + dated provenance event on the timeline; re-scanning does not duplicate + (dedup on normalized name + date against already-logged source events); + malformed markers (bad date shape, missing comma) are reported explicitly + and skipped; notes without markers produce no events; parsing is purely + structural. +- **Depends on**: none. + +### Task Q2 - configurable FTS tokenizer (`t_618f7211`, p3) +- **Files**: `src/core/search/schema.ts` (tokenizer string assembled from + config), config keys (e.g. `search.fts_tokenizer.*`), validation, docs + note that changing config requires `o2b search reindex`, tests. +- **Acceptance**: with no config the generated schema keeps + `unicode61 remove_diacritics 2` byte-identically; valid config changes the + tokenizer clause after reindex; invalid tokenizer options reject with a + typed error listing allowed values; the CJK trigram path is untouched + (its tests stay green); no implicit reindex. +- **Depends on**: none. + +### Task Q3 - graph-degree predicates in filter DSL (`t_9bee8f0b`, p3) +- **Files**: `src/core/search/property-filter.ts` (count predicates over + backlinks/outlinks), degree lookup via existing + `src/core/brain/link-graph/graph-index.ts` data, `src/cli/search.ts` and + MCP filter surface, tests. +- **Acceptance**: filters can select notes by backlink/outlink count with + `=`, `!=`, `>`, `>=`, `<`, `<=` (orphans `= 0`, hubs `>= N`); results match + the graph index degree data; invalid predicate syntax rejects with a typed + error; queries without degree predicates behave byte-identically. +- **Depends on**: none. + +### Task O2 - doctor repair mode (`t_bd6cc4cb`, p3, diagnostics anchor) +- **Files**: diagnostics-signal model (issue class + detector + optional + fixer + next-command hint) co-located with `src/core/brain/doctor.ts`, + `--repair` (dry-run preview default) and `--apply` on the doctor CLI/MCP, + fixers only for issue classes doctor already detects (orphaned references, + WAL gaps), typed event per applied fix, tests. +- **Acceptance**: `doctor --repair` previews planned fixes without writing; + `--repair --apply` performs them and logs one typed event per fix; + re-running after apply is a no-op (idempotent); `--strict` and plain + doctor remain read-only and byte-identical; unfixable issue classes state + so explicitly. +- **Depends on**: none. + +### Task O3 - unified operator status snapshot (`t_9f9c5466`, p3) +- **Files**: snapshot verb (CLI + MCP) composing existing signal sources + (health, doctor, hygiene, stale scan, review candidates, active profile, + state-file health) through the diagnostics-signal model, per-problem + next-command hint rendering, tests. +- **Acceptance**: one command prints a consolidated readable snapshot with + counts, stale/orphaned, review queue depth, active profile, and state + health; every problem line carries the exact next command to run (hint + supplied by the signal definition, not hardcoded in the formatter); a + healthy vault prints a compact all-clear; snapshot performs reads only. +- **Depends on**: O2 (diagnostics-signal model). + +### Task L - docs, CHANGELOG, version bump +- **Files**: `README.md`, `CHANGELOG.md` (`## [1.34.0]` + link reference), + `docs/cli-reference.md`, `docs/mcp.md`, `package.json` 1.34.0 + + `bun run scripts/sync-version.ts`. +- **Acceptance**: one CHANGELOG entry covers all eleven units; + `bun run sync-version:check` passes; README and reference docs cover the + new surfaces. +- **Depends on**: all previous tasks. diff --git a/docs/brainstorm/source-pipeline-integrity/variants.md b/docs/brainstorm/source-pipeline-integrity/variants.md new file mode 100644 index 00000000..bb5d9f26 --- /dev/null +++ b/docs/brainstorm/source-pipeline-integrity/variants.md @@ -0,0 +1,93 @@ +# Source pipeline integrity and operator tooling - variant audit trail + +Consultant: Claude Code (`claude -p`), prompt in `cli-output/prompt.md`, raw +output in `cli-output/claude.md`. Three variants were requested for the wave +architecture as a whole (where shared abstractions live), not for the worth +of individual units. The fallback consultant (Codex) was not invoked because +the primary run succeeded with three parseable variants. + +## Variant 1: Two shared kernels, cluster-local units + +- **Approach**: extract exactly two new shared abstractions, each riding in + with the first unit that needs it - a gitignore-style path-scope engine + (`src/core/fs/ignore.ts`) introduced by the hygiene-scan unit and consumed + by ingest scoping, and a diagnostics-signal model (issue class + detector + + optional fixer + next-command hint) introduced by `doctor --repair` and + consumed by the status snapshot. Everything else lands as local changes to + its existing module, in four clusters with a short dependency spine. +- **Trade-offs**: + - Pro: shared logic gets exactly one home without any infrastructure-only + commit; each abstraction ships inside a feature commit with tests. + - Pro: maximal prefix-shippability; a stalled wave leaves `main` + releasable after any unit. + - Pro: byte-identical opt-out is easy; nothing changes uninvoked. + - Con: the ingest path stays four separate touch points rather than one + explicit pipeline; a future wave may want that consolidation. + - Con: the diagnostics model shape is set by the repair unit and may need + a small extension when the snapshot unit arrives. +- **Complexity**: medium +- **Risk**: low + +## Variant 2: Source-pipeline spine + +- **Approach**: treat the five intake units as stages of one explicit + discovery pipeline abstraction (`SourcePipeline` context with typed hooks: + scope, gate, pre-extract, dispatch, reconcile); operator tooling reads a + shared health-signal registry; remaining units stay peripheral. +- **Trade-offs**: + - Pro: strongest conceptual match to the wave theme; one choke point for + what enters the vault. + - Pro: reconciliation and gating become trivially testable pipeline + stages. + - Con: the pipeline scaffold is an infrastructure commit the repo's + conventions discourage, and it forces refactoring `ingest.ts` / + `batch-plan.ts` before any user-visible unit ships; a stalled wave + leaves a half-migrated pipeline. + - Con: highest risk to byte-identical opt-out, since existing ingest is + rerouted through new machinery even when no new flag is set. + - Con: hygiene importing from `brain/ingest` inverts layering; the ignore + engine must live below both anyway. +- **Complexity**: large +- **Risk**: high + +## Variant 3: Eleven islands, convention-only sharing + +- **Approach**: every unit lands independently in its own existing module + with no new shared homes; ignore parsing lives inside `hygiene/` and + ingest imports it across layers; the snapshot re-formats six verbs' + outputs itself; fixers pair with doctor checks ad hoc. +- **Trade-offs**: + - Pro: cheapest to execute, easiest to parallelize, every commit small + and independently revertable. + - Pro: near-perfect prefix-shippability with zero scaffolding. + - Con: the ignore engine under `hygiene/` with `ingest/` importing it + violates one-directional layering, or gets duplicated. + - Con: the snapshot re-deriving signals from six verbs duplicates + presentation logic and drifts; hints get hardcoded per call site. + - Con: defers exactly the consolidation debt the post-v1.30.1 refactor + paid down. +- **Complexity**: small +- **Risk**: medium + +## Consultant recommendation + +Variant 1. "Variant 1 delivers the two abstractions this wave genuinely +reuses (ignore composition, diagnostics signals with hints) in their correct +one-home locations, while avoiding Variant 2's infrastructure-first commit +and its threat to the byte-identical opt-out and atomic-feature-commit +conventions. Unlike Variant 3, it keeps layering one-directional - the +ignore engine sits below both hygiene and ingest, and hints travel with +issue definitions instead of being duplicated in the snapshot. Its short +dependency spine (1->2, 9->10) preserves the constraint that a partially +completed wave still ships as a coherent prefix of v1.34.0." (quoted +verbatim from the consultant output, arrows ASCII-normalized) + +## Orchestrator decision + +Variant 1 is adopted without override. It is the same anchor-owned shared +abstraction pattern that succeeded in the v1.32.0 and v1.33.0 waves, applied +to this wave's shape: here the two shared pieces (path-scope engine, +diagnostics-signal model) each ship inside the first consuming feature +commit, keeping every commit release-visible and the layering +one-directional. The accepted ordering constraints (P1 before P2, O2 before +O3) are recorded in `plan.md`. diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 51a15d4a..61e7e5b5 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -262,6 +262,27 @@ o2b brain session-grep gains --since