feat: retrieval quality and context delivery (v1.37.0)

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "open-second-brain",
-  "version": "1.36.0",
+  "version": "1.37.0",
   "description": "Plugin-first second brain package for AI agents and humans.",
   "author": {
     "name": "Open Second Brain contributors"

.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "open-second-brain",
-  "version": "1.36.0",
+  "version": "1.37.0",
   "description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
   "skills": "./skills",
   "hooks": "./hooks/hooks.json",

CHANGELOG.md

@@ -5,6 +5,27 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.37.0] - 2026-07-19
+
+A retrieval quality and context delivery wave: nine units that make search answer relationship-shaped and summary-shaped questions deterministically, explain and plan retrieval without touching live policy, and put the right context in front of the agent while stale operational state stays out. Relationship queries traverse typed edges as a fourth RRF arm, dedup and search respect composite scopes, operational state moves to an overwrite-only lane with a retrieval-time staleness barrier, prompts gain a cadence-controlled navigation tier, an opt-in strict hook redirects the first raw vault read to the search index, and codegraph partnering covers every project in the workspace. Two shared seams carry the wave: a composite scope-key module and an index-admission predicate. The kernel still calls no LLM, and every new surface is byte-identical when its flag or param is omitted.
+
+### Added
+
+- Typed-edge relational retrieval arm: a vocabulary-driven relational-query parser (schema-pack `link_types` with subset validation, no natural-language word lists) and bounded depth-2 typed-edge fan-out join hybrid search as a fourth RRF arm behind `search_relational_arm_enabled` (env `OPEN_SECOND_BRAIN_SEARCH_RELATIONAL_ARM`, default off, byte-identical when disabled); RRF and dedup keys carry source identity through the shared `rrfKey`, and the query cache scopes by canonical source-set key.
+- Composite scope keys with per-scope dedup and search scoping: one shared module owns the source-identity and owner/session/project scope vocabulary; page dedup folds the scope in additively (scopeless writes keep their existing keys, reruns over old rows collapse nothing new), and `brain_search` gains optional `session_scope` and `project_scope` filters that leave unfiltered results byte-identical.
+- Exact-state lane with a retrieval-time staleness barrier: `o2b brain state set|get|list|clear` maintains an overwrite-only, per-aspect operational-state lane under `Brain/state/`, excluded from FTS, vector, and graph layers by a new index-admission predicate (which defaults to admit, so no existing content leaves the index), while a retrieval barrier drops superseded exact-state rows from recall; `Brain/pinned.md` deliberately remains a searchable scratchpad, and the lane is the supported home for current-state values.
+- Deterministic summary-search router: query plans carry a `surface` verdict that routes structurally summary-shaped questions (source-targeted, artifact-kind, summary-typed pages) to the summary surface; non-summary routing stays byte-identical, the verdict never enters the plan hash or ranking, and the retrieval skill names the intended surface explicitly.
+- Per-store reranker fit check: `o2b search rerank-fit` samples recorded demand-log queries, computes the correlation between reranker scores and the base retrieval signal, and reports fits, out-of-domain, or inverted verdicts with a disable-or-swap recommendation; strictly read-only (probes disable cache and self-heal) and explicitly inapplicable on rerankerless vaults.
+- Shadow-only retrieval plan advisor: the `brain_retrieval_plan` MCP tool composes query intent and weights, impact-per-token density allocation, the calibrated token-impact ledger, and observed route latency into a read-only per-question plan with source strategy, token-budget allocation, graph-expansion advice, reliability with p95 latency, and a marginal-value stop that tightens with latency; the tool exposes no mutating parameters.
+- Tiered context injection with a cadence-controlled navigation tier: an additive navmap (top hubs by link degree plus vault size, neutralized and fenced as untrusted) injects on `UserPromptSubmit` only when the cadence is due, behind `nav_tier_enabled` (env `OPEN_SECOND_BRAIN_NAV_TIER_ENABLED`, default off) with `nav_tier_cadence_minutes` (env `OPEN_SECOND_BRAIN_NAV_TIER_CADENCE_MINUTES`, default 30); every inclusion decision is audited with its reason and added character count, and the every-turn kernel stays byte-identical.
+- Opt-in strict PreToolUse orientation hook: with `hook_strict_enabled` (env `OPEN_SECOND_BRAIN_HOOK_STRICT_ENABLED`), the first raw vault-file read of a session is denied once with a redirect naming the brain search surface, then downgrades to a soft nudge; any brain query or search refreshes a short-lived orientation stamp that suppresses the block; detection is structural (the tool call's path resolving inside the vault root), and every failure path fails open.
+- Workspace-wide codegraph partnering: `checkCodegraph` aggregates status across every discovered code project and threads `project_path` per query when the partnered codegraph supports it (feature-detected from the status usage text), degrading to the previous single-project behavior with an explicit note when it does not; single-project workspaces are byte-identical.
+
+### Changed
+
+- Hook stamps (`osb.nav_tier.*`, `osb.oriented.*`) live in a minimal per-session file-backed store under `.open-second-brain/hook-state/` with epoch-ms expiry; missing, malformed, or expired stamps read as absent and never throw.
+- The MCP tool surface grows from 107 to 108 tools (`brain_retrieval_plan`).
+
 ## [1.36.0] - 2026-07-19
 
 A knowledge intake and consolidation wave: eight units that carry content from the operator's pocket into the vault and turn what accumulates into trustworthy insight. Phone captures arrive through an inbound Telegram bot and drain through a report-then-apply routing pass, keyed web-research providers feed the citation-constrained pipeline, facts roll up into higher tiers on pure counters, entities gain diarized profiles with a stated-vs-evidenced gap, synthesis findings carry causal context and decomposed confidence behind an evidence-identity gate, the link graph gets a capped idempotent repair lane with efficacy holdouts, and skill proposals pass a deterministic verifier before reaching the pending queue. Two shared seams carry the wave: a capture-note contract and a keyed external-fetch helper. The kernel still calls no LLM, and every new surface is byte-identical when unused.
@@ -6602,6 +6623,7 @@ plugin config (vault field)`, and exits with a clear
 - Sandbox vault and plugin manifest fixtures for tests.
 - GitHub release workflow for tag-based and manually dispatched releases.
 
+[1.37.0]: https://github.com/itechmeat/open-second-brain/compare/v1.36.0...v1.37.0
 [1.36.0]: https://github.com/itechmeat/open-second-brain/compare/v1.35.0...v1.36.0
 [1.35.0]: https://github.com/itechmeat/open-second-brain/compare/v1.34.0...v1.35.0
 [1.34.0]: https://github.com/itechmeat/open-second-brain/compare/v1.33.0...v1.34.0

README.md

@@ -8,9 +8,9 @@ Open Second Brain plugs into [Hermes Agent](https://github.com/NousResearch/herm
 
 ## What is new
 
-Open Second Brain 1.36.0 is a knowledge intake and consolidation release. Content now reaches the vault from the operator's pocket: an inbound Telegram bot (explicit long-poll runner, chat allowlist, `/catchup`) lands each message as a staged capture with provenance, and `o2b brain inbox-drain` classifies every capture structurally and routes it to a source page, a note, or an obligation with a per-item report, dry-run first. Keyed Brave and Tavily providers plus a full-page extract step feed the citation-constrained research pipeline and stay byte-identical without keys. What accumulates now consolidates: facts roll up into higher tiers on pure counters inside the dream pass, `o2b brain diarize` distills everything known about an entity into a profile with a stated-vs-evidenced gap, synthesis findings carry causal context and decomposed confidence behind an evidence-identity gate that counts every exclusion, `o2b brain repair-lane` densifies the link graph with a capped, idempotent, dry-run-first lane whose holdout harness proves graph lift and fails on dangling or unhydrated targets, and skill proposals pass a deterministic verifier, carry versions, and merge same-name collisions before a human ever reviews them. The kernel still calls no LLM.
+Open Second Brain 1.37.0 is a retrieval quality and context delivery release. Search now answers relationship questions natively: a typed-edge relational arm (opt-in, vocabulary-driven, bounded depth 2) joins hybrid RRF, query plans route summary-shaped questions to the summary surface deterministically, and dedup plus search respect composite owner/session/project scopes so identical text in two contexts is never wrongly collapsed. Retrieval also explains itself: `o2b search rerank-fit` tells you when a reranker is out-of-domain or score-inverted on your vault, and the read-only `brain_retrieval_plan` MCP tool lays out per question the source strategy, token allocation, reliability with p95 latency, and a marginal-value stop, without touching live ranking policy. On the delivery side, a cadence-controlled navigation tier can inject a fenced vault map on top of the always-on kernel, an opt-in strict hook redirects the first raw vault read of a session to the search index and then steps back, operational state moves into an overwrite-only `o2b brain state` lane that is excluded from semantic layers and barred from resurfacing stale values through recall, and codegraph partnering covers every project in the workspace instead of the first one found. The kernel still calls no LLM.
 
-Previous release, 1.35.0, was a trusted recall and memory write surface wave: a bounded audited recall brief on every prompt, a retrieval trust gate with per-pack receipts, relation-only supersede fade, the full MCP note write lifecycle, and fail-fast doctor readiness probes. Details live in the [CHANGELOG](CHANGELOG.md).
+Previous release, 1.36.0, was a knowledge intake and consolidation wave: an inbound Telegram capture bot with a report-then-apply inbox drain, keyed Brave and Tavily research providers, a count-triggered fact rollup ladder, subject diarization, an evidence-identity gate on synthesis findings, a capped idempotent link-graph repair lane, and a deterministic skill-proposal verifier. Details live in the [CHANGELOG](CHANGELOG.md).
 
 ## Why
 

docs/brainstorm/retrieval-quality-and-context-delivery/cli-output/claude.md

@@ -0,0 +1,35 @@
+### Variant 1: Platform-first extraction
+- **Approach**: Extract all five candidate seams as shared modules before any feature task lands: a `scope-key` module (owner/session/project composite keys), a read-only `retrieval-diagnostics` composition layer wrapping query-plan/density/reliability/latency signals, a generalized session-state store in `hooks/lib`, a unified index-admission policy in the walker, and the RRF source-identity key helper. The nine tasks then consume these as libraries, ordered infra-first (seams, then 8, 1, 7, then the rest).
+- **Trade-offs**:
+  - Pro: source identity and scope keying each have exactly one implementation from day one; tasks 1, 7, 8 cannot drift on key semantics.
+  - Pro: the diagnostics layer guarantees 3 and 4 stay shadow-only by construction (the layer exposes no mutating handles).
+  - Con: two of five seams (diagnostics composition, generalized hook state) have only two consumers each and no third in sight; this is speculative abstraction the "no stubs, no do-nothing surfaces" culture pushes against.
+  - Con: infra commits precede any user-visible behavior, inflating an already nine-commit PR and making the byte-identical-default review harder (reviewers must check unused-yet code paths).
+  - Con: getting the scope-key vocabulary right before task 8's dedup-migration questions are answered risks a mid-wave redesign of the very module everything depends on.
+- **Complexity**: large
+- **Risk**: medium
+
+### Variant 2: Two hard seams, rest conventions
+- **Approach**: Extract only the seams where a second divergent implementation would be a correctness bug: (a) one composite-key module providing both the RRF/dedup source-identity key (task 1's federation hardening) and the scope-key vocabulary (task 8's per-namespace dedup and search filters, plus task 1's scope-aware seed resolution); (b) one index-admission predicate at the walker/indexer touch point, owned by task 7's lane exclusion and consulted by task 8's search scoping. Everything else stays a documented convention: 3 and 4 each read the existing signal modules directly (shadow-only enforced by API choice and tests, not a layer), and 5 and 6 store their stamps as namespaced keys in the existing `hooks/lib` session state. Ordering: key module rides in with task 8, task 1 consumes it next; admission predicate rides in with task 7; tasks 2, 3, 4, 5, 6, 9 are order-independent and can interleave.
+- **Trade-offs**:
+  - Pro: each extraction ships inside the atomic commit of its first consumer, matching the one-commit-per-task convention with no bare infra commits.
+  - Pro: the two extracted seams are exactly the ones where drift is silent and dangerous (a key mismatch collapses or fails to collapse results; a walker mismatch drops or leaks index rows); the convention seams fail loudly in review instead.
+  - Pro: smallest total surface for the byte-identical-default audit; diagnostics tasks 3 and 4 remain independent and can be reviewed or even dropped without unwinding shared code.
+  - Con: 3 and 4 will duplicate some signal-plumbing (sampling queries, reading the reliability ledger); a future third diagnostic surface would trigger a refactor.
+  - Con: hook-state conventions rely on discipline (key naming, TTL semantics) rather than types; a collision between the "recently oriented" stamp and nav-cadence state is possible if the convention is under-specified.
+- **Complexity**: medium
+- **Risk**: low
+
+### Variant 3: Extract-on-second-use, ordering-driven
+- **Approach**: No upfront extraction at all; sequence the wave so the first consumer of each candidate seam lands early with inline code, and the second consumer performs the extraction as part of its own commit (rule of two). Concretely: 7 ships walker exclusion inline, then 8 hoists it into a shared predicate when adding scope filters; 1 ships its RRF key inline, then 8 hoists the key helper for dedup; 3 ships inline sampling, then 4 hoists a shared diagnostics reader; 6 ships its stamp inline, then 5 hoists cadence state.
+- **Trade-offs**:
+  - Pro: zero speculative abstraction; every shared module is proven by two real call sites at the moment it is born.
+  - Pro: early tasks are maximally simple and can merge-review independently; nothing blocks on a shared-module design debate.
+  - Con: second-consumer commits are no longer atomic to their own task; they mix "implement task 8" with "refactor tasks 1 and 7", violating the one-atomic-commit-per-task convention or forcing extra refactor commits inside the PR.
+  - Con: within a single PR the churn is pure waste; code written in commit 3 is rewritten in commit 6 before any reviewer sees it stable, and the byte-identical regression tests for 7 must be re-verified after 8's hoist.
+  - Con: the key-semantics seam (source identity plus scope) is precisely where two independently written inline versions are most likely to disagree subtly before unification.
+- **Complexity**: medium
+- **Risk**: medium
+
+### Recommended: Variant 2
+**Rationale**: The wave ships as one PR with one atomic commit per task, so both the bare-infra commits of Variant 1 and the intra-PR rewrite churn of Variant 3 fight the repo's own conventions, while Variant 2 lets each shared module ride in with its first real consumer. It extracts exactly the two seams where divergence is a silent correctness bug (composite ranking/dedup keys, walker index admission) and leaves the advisory and hook-state seams as test-enforced conventions, which best serves the byte-identical-default and shadow-only constraints with the smallest review surface. Ordering falls out naturally: 8 then 1 around the key module, 7 before 8's search scoping at the walker, and the remaining six tasks stay freely schedulable.