feat: source pipeline integrity and operator tooling (v1.34.0)

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "open-second-brain",
-  "version": "1.33.0",
+  "version": "1.34.0",
   "description": "Plugin-first second brain package for AI agents and humans.",
   "author": {
     "name": "Open Second Brain contributors"

.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "open-second-brain",
-  "version": "1.33.0",
+  "version": "1.34.0",
   "description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
   "skills": "./skills",
   "hooks": "./hooks/hooks.json",

CHANGELOG.md

@@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.34.0] - 2026-07-18
+
+A source pipeline integrity and operator tooling wave: eleven units that make vault intake trustworthy end to end (scoped and gated discovery, reconciled dispatch, deterministic pre-extraction, citation provenance), extend the query surface (configurable FTS tokenizer, graph-degree predicates), and give the operator a repairing doctor plus one consolidated status snapshot. Two shared kernels carry the wave: a gitignore-semantics path-scope engine and a diagnostics-signal registry whose next-command hints travel with the issue definitions. Everything stays deterministic (the kernel calls no LLM) and every new surface is byte-identical when unused.
+
+### Added
+
+- Nested `.gitignore` support in the hygiene file scan through a new path-scope engine (`src/core/fs/ignore.ts`): root and nested ignore files compose with git semantics (a deeper file scopes its own subtree, a nearer `!` re-include wins, `.git/info/exclude` participates); repositories without ignore files scan exactly as before and malformed patterns warn explicitly.
+- Source ingest scoping: `--src-subpath` and `--exclude` on `o2b brain batch-plan` (and `src_subpath` / `exclude` on `brain_ingest_batch_plan`) restrict a monorepo ingest to one subtree with gitignore-style excludes through the same shared engine; a subpath escaping the source root rejects with a typed error.
+- The schema `extractable` flag is enforced during page discovery: with a non-empty allowlist, pages whose `schema_type` is not listed are skipped before extraction, reported with a reason on the plan (`skipped_non_extractable`) and logged; an empty allowlist keeps today's behavior byte for byte and untyped pages are never gated.
+- Deterministic no-LLM code-structure pre-extraction (`o2b brain pre-extract`, `pre_extract` on `brain_ingest_source`): classes, functions, imports, and inheritance edges from TypeScript, JavaScript, and Python sources as JSON entity/edge seeds handed to the agent ingest step; unknown languages report `extracted: false` with a reason, never a fake empty success.
+- Dispatched-vs-ingested reconciliation (`o2b brain batch-plan --reconcile`, `reconcile` on `brain_ingest_batch_plan`): diffs a plan's dispatched set against checkpoint completions and names every silently-lost source; a complete plan reports an explicit empty gap; read-only and idempotent, a report rather than a retry.
+- Inline citation promotion (`o2b brain scan-citations`): structural `[Source: <name>, YYYY-MM-DD]` markers in note prose become dated `source-citation` events on the temporal timeline, stamped at the citation date and deduplicated on normalized name plus date; malformed markers are reported and skipped (`--strict` exits nonzero).
+- Configurable FTS tokenizer: `search_fts_diacritics` (`0 | 1 | 2`) and `search_fts_stemmer` (`none | porter`) assemble the FTS5 tokenizer clause from validated allowlists; invalid values reject with a typed error naming the allowed set, changing the config requires an explicit `o2b search reindex`, and the CJK trigram path is untouched.
+- Graph-degree cardinality predicates in the search filter DSL: `--degree` on `o2b search` (and `degree` on `brain_search`) filters by backlink and outlink counts with `=`, `!=`, `>`, `>=`, `<`, `<=` (orphans `backlinks=0`, hubs `outlinks>=N`), backed by directed degree data on the existing link-graph index.
+- Doctor repair mode (`o2b brain doctor --repair`, `repair` / `apply` on `brain_doctor`): previews targeted fixes for doctor-detected issue classes (dangling workrun checkpoints, dead evidence links) without writing; `--repair --apply` performs them idempotently and logs one typed `doctor-repair` event per fix; structural lifecycle links and append-only audit history are reported needs-review or unfixable rather than half-fixed; plain doctor and `--strict` stay read-only.
+- Unified operator status snapshot (`o2b brain status`, MCP `brain_status`): doctor, semantic health, hygiene, stale scan, review queue depth, active profile, and state-file health in one readable view; every problem line carries the exact next command supplied by the shared diagnostics-signal registry; a healthy vault prints a compact all-clear.
+- Clean exit on early-closed stdout pipes: `o2b ... | head` (and `vault-log`) exits 0 on EPIPE, while any other stdout error now fails loudly instead of being silently swallowed.
+
+### Changed
+
+- The MCP tool surface grows from 102 to 103 tools (`brain_status`); `brain_ingest_batch_plan`, `brain_ingest_source`, `brain_doctor`, and `brain_search` gain the optional params listed above and stay byte-identical when the params are omitted.
+- The hygiene scan replaces its static skip-dir list with gitignore-aware composition via the shared path-scope engine.
+- Doctor diagnostics flow through a shared diagnostics-signal registry (issue class, detector, optional fixer, next-command hint) consumed by both `doctor --repair` and `o2b brain status`.
+
 ## [1.33.0] - 2026-07-18
 
 A belief lifecycle and decision memory wave: eleven units that give memories a full lifecycle (tombstone, supersede, temporal replacement, tension triage) and make decisions first-class, auditable citizens of the brain (records, ratings, commitment tiers, change receipts, governed recall). Everything is deterministic (the kernel still calls no LLM), every mutation is logged or receipted, and vaults that do not opt in behave byte-identically.
@@ -6532,6 +6556,7 @@ plugin config (vault field)`, and exits with a clear
 - Sandbox vault and plugin manifest fixtures for tests.
 - GitHub release workflow for tag-based and manually dispatched releases.
 
+[1.34.0]: https://github.com/itechmeat/open-second-brain/compare/v1.33.0...v1.34.0
 [1.33.0]: https://github.com/itechmeat/open-second-brain/compare/v1.32.0...v1.33.0
 [1.32.0]: https://github.com/itechmeat/open-second-brain/compare/v1.31.0...v1.32.0
 [1.31.0]: https://github.com/itechmeat/open-second-brain/compare/v1.30.1...v1.31.0

README.md

@@ -8,7 +8,7 @@ Open Second Brain plugs into [Hermes Agent](https://github.com/NousResearch/herm
 
 ## What is new
 
-Open Second Brain 1.33.0 is a belief lifecycle and decision memory release. Memories now have a full lifecycle: an idempotent tombstone and supersede operation spans every memory type, facts can be replaced atomically at one instant with gap-free half-open validity intervals, a persisted claim graph answers "what is true now, what used to be true, what replaced it, what contests it" in one query, and injection prefers the tip of a supersedes chain while recall points superseded hits at their replacement. Decisions become first-class: recorded with assumption and review date (which opens a review obligation), rated and compared, tiered by commitment (exploring, leaning, decided, locked), receipted on every change with before/after/evidence trails, and resurfaced verbatim into matching sessions under per-session caps and spacing. Detected contradictions persist as triageable tension objects that warn at injection time until resolved, and transcript turns keep their authored time all the way into search ranking and time-bounded queries. The kernel still calls no LLM.
+Open Second Brain 1.34.0 is a source pipeline integrity and operator tooling release. Getting sources into the vault is now trustworthy end to end: ingest can be scoped to one monorepo subtree with gitignore-style excludes, the hygiene scan honors nested `.gitignore` files with git semantics, the schema `extractable` flag actually gates page discovery, a deterministic no-LLM pre-pass extracts code structure (classes, functions, imports, inheritance) as seeds before agent ingest, and every batch plan can be reconciled so a source an agent silently dropped is named instead of lost. Prose citations like `[Source: name, YYYY-MM-DD]` become dated provenance events on the temporal timeline, the FTS tokenizer becomes configurable for non-English vaults, and search gains graph-degree predicates for orphan and hub queries. On the operator side, `o2b brain doctor --repair` turns diagnostics into guarded one-command fixes, `o2b brain status` consolidates every health signal into one snapshot with an exact next command per problem, and piping any CLI into `head` finally exits clean. The kernel still calls no LLM.
 
 ## Why
 

docs/brainstorm/source-pipeline-integrity/cli-output/claude.md

@@ -0,0 +1,38 @@
+### Variant 1: Two shared kernels, cluster-local units
+
+- **Approach**: Extract exactly two new shared abstractions, each riding in with the first unit that needs it: a gitignore-style path-scope engine (`src/core/fs/ignore.ts`: nested ignore-file composition, nearer-`!`-wins, `.git/info/exclude` layering) introduced by Unit 1 and consumed by Unit 2's `--src-subpath`/`--exclude`; and a diagnostics-signal model (issue class + detector + optional fixer + next-command hint) introduced by Unit 9 and consumed by Unit 10's snapshot. Everything else lands as local changes to its existing module, organized into four clusters: scope/gate (1→2→3), ingest integrity (5, 4), provenance/query (6, 7, 8), operator surface (11, 9→10). The dependency spine is short - only 1→2 and 9→10 are hard edges; ship order 11, 1, 2, 3, 5, 4, 6, 7, 8, 9, 10 keeps every prefix coherent.
+- **Trade-offs**:
+  - Pro: shared logic gets exactly one home (ignore matching, issue/hint model) without any infrastructure-only commit - each abstraction ships inside a feature commit with its tests.
+  - Pro: maximal prefix-shippability; a stalled wave still leaves `main` releasable after any unit.
+  - Pro: byte-identical opt-out is easy - the ignore engine replaces the static skip-dir list only where Unit 1 wires it; nothing else changes uninvoked.
+  - Con: the ingest path (Units 2, 3, 5, 4) stays four separate touch points in `ingest.ts`/`batch-plan.ts` rather than one explicit pipeline; a future wave may still want that consolidation.
+  - Con: the diagnostics model's shape is decided by Unit 9's needs and may need a small extension when Unit 10 arrives.
+- **Complexity**: medium
+- **Risk**: low
+
+### Variant 2: Source-pipeline spine
+
+- **Approach**: Treat Units 1-5 as stages of one explicit discovery pipeline abstraction in `src/core/brain/ingest/`: scope (ignore engine + subpath/exclude) → gate (`extractable` check) → pre-extract (deterministic code-structure pass) → dispatch → reconcile, each stage a typed hook on a shared `SourcePipeline` context. Operator tooling (9, 10) similarly reads a shared health-signal registry, and hygiene scan (Unit 1) becomes another consumer of the same scope stage. Units 6, 7, 8, 11 remain peripheral standalone changes.
+- **Trade-offs**:
+  - Pro: strongest conceptual match to the wave theme - one choke point for "what enters the vault and how," which future ingest features slot into.
+  - Pro: reconciliation (Unit 4) and gating (Unit 3) become trivially testable as pipeline stages rather than call-site patches.
+  - Con: the pipeline scaffold is effectively an infrastructure commit, which the repo's conventions discourage, and it forces refactoring `ingest.ts`/`batch-plan.ts` before any user-visible unit ships - a stalled wave leaves a half-migrated pipeline.
+  - Con: highest risk to the byte-identical opt-out constraint, since existing ingest flow is rerouted through new machinery even when no new flag is set.
+  - Con: hygiene scan (Unit 1) importing from `brain/ingest` inverts sensible layering; the ignore engine would need to live below both anyway, eroding the "one spine" elegance.
+- **Complexity**: large
+- **Risk**: high
+
+### Variant 3: Eleven islands, convention-only sharing
+
+- **Approach**: Every unit lands independently in its own existing module with no new shared homes: Unit 1 implements ignore parsing inside `hygiene/`, Unit 2 imports that parser directly (a hygiene→ingest reach-across), Unit 10 calls the existing verb functions (`brain_health`, `brain_doctor`, etc.) and formats their outputs itself, and Unit 9 pairs fixers with doctor checks ad hoc inside `doctor.ts`. Ordering is nearly free - only Unit 2's import of Unit 1's parser is a hard edge.
+- **Trade-offs**:
+  - Pro: cheapest to execute and the easiest to parallelize across agents; every commit is small, atomic, and independently revertable.
+  - Pro: near-perfect prefix-shippability with zero scaffolding to leave half-finished.
+  - Con: the ignore engine living under `hygiene/` while `ingest/` imports it violates the one-directional-layering direction, or else gets duplicated - either way conflicting with "shared logic gets exactly one home."
+  - Con: Unit 10 re-deriving and re-formatting signals from six verbs duplicates presentation logic and drifts as those verbs evolve; hints get hardcoded per call site instead of traveling with the issue.
+  - Con: defers exactly the consolidation debt the post-v1.30.1 refactor just paid down.
+- **Complexity**: small
+- **Risk**: medium
+
+### Recommended: Variant 1
+**Rationale**: Variant 1 delivers the two abstractions this wave genuinely reuses (ignore composition, diagnostics signals with hints) in their correct one-home locations, while avoiding Variant 2's infrastructure-first commit and its threat to the byte-identical opt-out and atomic-feature-commit conventions. Unlike Variant 3, it keeps layering one-directional - the ignore engine sits below both hygiene and ingest, and hints travel with issue definitions instead of being duplicated in the snapshot. Its short dependency spine (1→2, 9→10) preserves the constraint that a partially completed wave still ships as a coherent prefix of v1.34.0.