diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b8a587..a487481 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,13 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning: [S ## [Unreleased] +### Added +- **Automatic per-turn recall on Claude Code.** A `UserPromptSubmit` hook runs a + hybrid recall against each substantive prompt and injects the hits as context, + closing the gap where the `recall` tool was almost never invoked. New + `cairn recall-hook` command; configurable via `auto_recall` / `auto_recall_k` / + `auto_recall_scope` (default on). Plugin `0.4.0`. + ### Fixed - **Claude Code plugin (0.3.1): fresh install no longer fails on first run.** The SessionStart/SessionEnd/PreCompact hooks passed `${user_config.vault_path}` as a command diff --git a/README.md b/README.md index 0986221..5c85fbd 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,21 @@ codex plugin add agentcairn@agentcairn On install you pick a vault path (default `~/agentcairn`); it's **auto-created** on the first session — no Obsidian setup required. From then on agentcairn surfaces relevant memory at the start of each session, distills each session into your vault, and gives you `/agentcairn:recall`, `/remember`, `/memory`, `/savings`, and `/ingest`. Nothing to pip-install — the plugin runs the published package via `uvx`. +**Automatic recall (Claude Code).** On every substantive prompt, the plugin runs +a hybrid recall against what you just asked and injects the most relevant +memories as context for that turn — not just the recency digest shown at session +start. Trivially-short prompts (e.g. "yes", "go") are skipped. It is fail-open: +if anything goes wrong it injects nothing and never blocks your prompt. + +Configure it in `~/.agentcairn/config.toml` (flat top-level keys; env vars +`CAIRN_AUTO_RECALL`, `CAIRN_AUTO_RECALL_K`, `CAIRN_AUTO_RECALL_SCOPE` override the file): + +```toml +auto_recall = true # master on/off (default: true) +auto_recall_k = 3 # memories injected per prompt +auto_recall_scope = "all" # "all" (boost, non-lossy) or "project" (hard filter) +``` + > Not on Claude Code or Codex? agentcairn is also a standalone MCP server + CLI for any host — see [Using it directly](#using-it-directly). ## How it works diff --git a/docs/plans/2026-06-29-auto-recall-userpromptsubmit.md b/docs/plans/2026-06-29-auto-recall-userpromptsubmit.md new file mode 100644 index 0000000..bb3e70e --- /dev/null +++ b/docs/plans/2026-06-29-auto-recall-userpromptsubmit.md @@ -0,0 +1,760 @@ +# Auto per-turn recall (UserPromptSubmit) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make agentcairn run a hybrid recall against each substantive Claude Code prompt and inject the hits as context, closing the gap where recall was never triggered. + +**Architecture:** A thin Python command `cairn recall-hook` holds all logic (config gate, trivial-prompt gate, hybrid recall, formatting, fail-open). A `UserPromptSubmit` hook calls a small shell wrapper that execs it synchronously (its stdout is the injected context). SessionStart pre-warms the embedder. The SessionStart recency digest is unchanged. + +**Tech Stack:** Python 3 (Typer CLI, DuckDB-backed `search()`), `pytest` + Typer `CliRunner` + `FakeEmbedder` for hermetic tests, POSIX `sh` plugin hooks. + +## Global Constraints + +- **Fail-open, always exit 0:** `recall-hook` and `user-prompt-submit.sh` must NEVER raise, block, or exit non-zero — a `UserPromptSubmit` hook that exits non-zero *blocks the prompt*. Every failure path returns `""` / emits nothing. +- **Flat config keys:** new knobs are top-level `config.toml` keys (`auto_recall`, `auto_recall_k`, `auto_recall_scope`) → `CAIRN_AUTO_RECALL{,_K,_SCOPE}`. The loader does not read `[section]` tables. +- **Rerank OFF** on the recall-hook hot path (latency). +- **Plugin entries use `${CLAUDE_PLUGIN_ROOT}`**, never `${user_config.*}` — the guard test `test_hooks_do_not_hardfail_on_unset_vault_path` asserts `"${user_config.vault_path}" not in json.dumps(hooks)` over the whole file. Resolve the vault inside the script from `$CLAUDE_PLUGIN_OPTION_VAULT_PATH`. +- **Hermetic tests:** build indexes with `FakeEmbedder` (`get_embedder("fake")`) / `--embedder fake`; never load real fastembed in tests. +- **Plugin tests run outside default `testpaths`:** run them explicitly with `uv run pytest plugin/tests/test_plugin.py`. +- **Commit trailer:** end every commit message with the repo's standard trailer: + ``` + Co-Authored-By: Claude Opus 4.8 + Claude-Session: https://claude.ai/code/session_01UyeqgET1ZMb59jeZLiREvD + ``` +- **Run the suite** with `uv run pytest` from the repo root. + +## File Structure + +- `src/cairn/config.py` — *modify*: add 3 `KNOBS` entries + 3 `resolve_*()` functions + one default constant. +- `src/cairn/recall_hook.py` — *create*: all auto-recall logic (pure units + `run()` orchestrator). +- `src/cairn/cli.py` — *modify*: add the `recall-hook` Typer command (thin stdin→`run()` shim). +- `plugin/scripts/user-prompt-submit.sh` — *create*: synchronous wrapper that execs `cairn recall-hook`. +- `plugin/scripts/session-start.sh` — *modify*: add a detached `cairn warm` on the warm path. +- `plugin/hooks/hooks.json` — *modify*: add the `UserPromptSubmit` entry. +- `plugin/.claude-plugin/plugin.json` — *modify*: version bump `0.3.1` → `0.4.0`. +- `tests/test_config_auto_recall.py` — *create*: resolver tests. +- `tests/test_recall_hook.py` — *create*: core-logic tests (direct `run()` + pure units). +- `tests/test_recall_hook_cli.py` — *create*: one CliRunner stdin-wiring test. +- `plugin/tests/test_plugin.py` — *modify*: assert the `UserPromptSubmit` wiring. +- `README.md`, `CHANGELOG.md` — *modify*: document auto-recall + config; changelog entry. + +--- + +### Task 1: Config knobs + resolvers + +**Files:** +- Modify: `src/cairn/config.py` (KNOBS tuple ~line 49-104; add resolvers near `resolve_consolidate` ~line 229) +- Test: `tests/test_config_auto_recall.py` (create) + +**Interfaces:** +- Produces: + - `resolve_auto_recall(env: Mapping[str, str] | None = None) -> bool` (default `True`) + - `resolve_auto_recall_k(env: Mapping[str, str] | None = None) -> int` (default `3`) + - `resolve_auto_recall_scope(env: Mapping[str, str] | None = None) -> str` (default `"all"`, lower-cased) +- Consumes: existing `parse_bool`, `cairn_env`, `Knob`, `KNOBS` from the same module. + +- [ ] **Step 1: Write the failing tests** + +Create `tests/test_config_auto_recall.py`: + +```python +# SPDX-License-Identifier: Apache-2.0 +from cairn.config import ( + resolve_auto_recall, + resolve_auto_recall_k, + resolve_auto_recall_scope, +) + + +def test_auto_recall_default_on(): + assert resolve_auto_recall(env={}) is True + + +def test_auto_recall_off(): + assert resolve_auto_recall(env={"CAIRN_AUTO_RECALL": "0"}) is False + assert resolve_auto_recall(env={"CAIRN_AUTO_RECALL": "false"}) is False + + +def test_auto_recall_bad_value_falls_back_true(): + assert resolve_auto_recall(env={"CAIRN_AUTO_RECALL": "maybe"}) is True + + +def test_auto_recall_k_default(): + assert resolve_auto_recall_k(env={}) == 3 + + +def test_auto_recall_k_override(): + assert resolve_auto_recall_k(env={"CAIRN_AUTO_RECALL_K": "5"}) == 5 + + +def test_auto_recall_k_bad_falls_back(): + assert resolve_auto_recall_k(env={"CAIRN_AUTO_RECALL_K": "lots"}) == 3 + + +def test_auto_recall_scope_default(): + assert resolve_auto_recall_scope(env={}) == "all" + + +def test_auto_recall_scope_override_lowercased(): + assert resolve_auto_recall_scope(env={"CAIRN_AUTO_RECALL_SCOPE": "Project"}) == "project" +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/test_config_auto_recall.py -v` +Expected: FAIL with `ImportError: cannot import name 'resolve_auto_recall'`. + +- [ ] **Step 3: Add the KNOBS entries** + +In `src/cairn/config.py`, inside the `KNOBS` tuple (after the `consolidate` `Knob`, before the closing `)`), add: + +```python + Knob( + "auto_recall", + "CAIRN_AUTO_RECALL", + "true", + "Auto-recall relevant memory before each substantive prompt (Claude Code).", + ), + Knob( + "auto_recall_k", + "CAIRN_AUTO_RECALL_K", + "3", + "How many memories auto-recall injects per prompt.", + ), + Knob( + "auto_recall_scope", + "CAIRN_AUTO_RECALL_SCOPE", + "all", + "Auto-recall scope: 'all' (boost, non-lossy) or 'project' (hard filter).", + ), +``` + +- [ ] **Step 4: Add the default constant + resolvers** + +In `src/cairn/config.py`, near the other `_DEFAULT_*` constants add: + +```python +_DEFAULT_AUTO_RECALL_K = 3 +``` + +After `resolve_consolidate` add: + +```python +def resolve_auto_recall(env: Mapping[str, str] | None = None) -> bool: + """Resolve auto-recall on/off: CAIRN_AUTO_RECALL env/file → True. + An unparseable value falls back to the default (True) rather than raising, + so a typo never disables recall silently or breaks a prompt.""" + if env is None: + env = cairn_env() + raw = env.get("CAIRN_AUTO_RECALL") + if raw is None: + return True + try: + return parse_bool(raw) + except ValueError: + return True + + +def resolve_auto_recall_k(env: Mapping[str, str] | None = None) -> int: + """Resolve auto-recall depth: CAIRN_AUTO_RECALL_K env/file → 3. + An unparseable value falls back to the default rather than raising.""" + if env is None: + env = cairn_env() + try: + return int(env.get("CAIRN_AUTO_RECALL_K") or _DEFAULT_AUTO_RECALL_K) + except ValueError: + return _DEFAULT_AUTO_RECALL_K + + +def resolve_auto_recall_scope(env: Mapping[str, str] | None = None) -> str: + """Resolve auto-recall scope: CAIRN_AUTO_RECALL_SCOPE env/file → 'all'.""" + if env is None: + env = cairn_env() + return (env.get("CAIRN_AUTO_RECALL_SCOPE") or "all").strip().lower() +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `uv run pytest tests/test_config_auto_recall.py -v` +Expected: PASS (8 passed). + +- [ ] **Step 6: Commit** + +```bash +git add src/cairn/config.py tests/test_config_auto_recall.py +git commit # message: "feat(config): auto_recall / _k / _scope knobs + resolvers" + trailer +``` + +--- + +### Task 2: `recall_hook` core module + +**Files:** +- Create: `src/cairn/recall_hook.py` +- Test: `tests/test_recall_hook.py` + +**Interfaces:** +- Consumes (Task 1): `resolve_auto_recall`, `resolve_auto_recall_k`, `resolve_auto_recall_scope`, `cairn_env`. +- Consumes (existing): `cairn.paths.index_for`, `cairn.paths.resolve_vault`, `cairn.embed.get_embedder`, `cairn.search.open_search`, `cairn.search.resolve_current_project`, `cairn.search.search` (returns `list[Hit]`, each `Hit` has `.permalink`, `.heading_path`, `.snippet`, `.score`). +- Produces (Task 3 consumes): + - `should_recall(prompt: str, env: Mapping | None = None) -> bool` + - `format_block(notes: list[dict]) -> str` + - `build_hook_output(block: str) -> dict` + - `run(stdin_text: str, *, vault=None, index=None, embedder_name="fastembed", env=None) -> str` + +- [ ] **Step 1: Write the failing tests** + +Create `tests/test_recall_hook.py`: + +```python +# SPDX-License-Identifier: Apache-2.0 +import json +from pathlib import Path + +from cairn.embed import get_embedder +from cairn.recall_hook import build_hook_output, format_block, run, should_recall +from tests.search.test_engine import build_index + + +def _idx(tmp_path) -> Path: + return Path(build_index(tmp_path, get_embedder("fake"))) + + +def test_should_recall_gate(): + assert should_recall("how do I brew coffee beans?", env={}) is True + assert should_recall("go", env={}) is False + assert should_recall(" yes ", env={}) is False + assert should_recall("how do I brew coffee?", env={"CAIRN_AUTO_RECALL": "0"}) is False + + +def test_format_block_empty_returns_empty(): + assert format_block([]) == "" + assert format_block([{"permalink": "x", "text": " "}]) == "" + + +def test_format_block_includes_permalink(): + block = format_block([{"permalink": "coffee", "text": "Arabica beans."}]) + assert block.startswith("## Relevant memories (agentcairn)") + assert "Arabica beans." in block + assert "[[coffee]]" in block + + +def test_build_hook_output_shape(): + assert build_hook_output("hi") == { + "hookSpecificOutput": { + "hookEventName": "UserPromptSubmit", + "additionalContext": "hi", + } + } + + +def test_run_injects_relevant_memory(tmp_path): + out = run( + json.dumps({"prompt": "how do I brew coffee beans?"}), + index=_idx(tmp_path), + embedder_name="fake", + env={}, + ) + assert out + data = json.loads(out) + assert data["hookSpecificOutput"]["hookEventName"] == "UserPromptSubmit" + assert "coffee" in data["hookSpecificOutput"]["additionalContext"].lower() + + +def test_run_skips_trivial_prompt(tmp_path): + out = run(json.dumps({"prompt": "go"}), index=_idx(tmp_path), embedder_name="fake", env={}) + assert out == "" + + +def test_run_disabled_via_env(tmp_path): + out = run( + json.dumps({"prompt": "how do I brew coffee beans?"}), + index=_idx(tmp_path), + embedder_name="fake", + env={"CAIRN_AUTO_RECALL": "0"}, + ) + assert out == "" + + +def test_run_no_index_is_silent(tmp_path): + out = run( + json.dumps({"prompt": "how do I brew coffee beans?"}), + index=tmp_path / "missing.duckdb", + embedder_name="fake", + env={}, + ) + assert out == "" + + +def test_run_malformed_stdin_is_silent(tmp_path): + out = run("not json at all", index=_idx(tmp_path), embedder_name="fake", env={}) + assert out == "" +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/test_recall_hook.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'cairn.recall_hook'`. + +- [ ] **Step 3: Implement the module** + +Create `src/cairn/recall_hook.py`: + +```python +# SPDX-License-Identifier: Apache-2.0 +"""UserPromptSubmit auto-recall. + +Runs a hybrid recall against the user's prompt and emits it as Claude Code +`additionalContext`. All logic lives here as small, testable units; the plugin +ships only a thin shell wrapper that execs the `cairn recall-hook` CLI command, +which delegates to `run()`. Every path is fail-open: `run()` never raises and +returns "" (inject nothing) on any problem.""" +from __future__ import annotations + +import json +from collections.abc import Mapping +from pathlib import Path + +from cairn import paths +from cairn.config import ( + cairn_env, + resolve_auto_recall, + resolve_auto_recall_k, + resolve_auto_recall_scope, +) +from cairn.embed import get_embedder +from cairn.search import open_search, resolve_current_project, search + +_DEFAULT_MIN_CHARS = 12 + + +def _min_chars(env: Mapping[str, str]) -> int: + try: + return int(env.get("CAIRN_AUTO_RECALL_MIN_CHARS") or _DEFAULT_MIN_CHARS) + except ValueError: + return _DEFAULT_MIN_CHARS + + +def should_recall(prompt: str, env: Mapping[str, str] | None = None) -> bool: + """True iff auto-recall is enabled and the prompt is substantive. + Skips trivially-short prompts ("yes", "go") — continuations where recall + adds noise, not signal.""" + if env is None: + env = cairn_env() + if not resolve_auto_recall(env): + return False + return len(prompt.strip()) >= _min_chars(env) + + +def format_block(notes: list[dict]) -> str: + """Render recalled notes into the injection markdown. Returns "" when there + is nothing to inject (empty list / all-blank texts) so callers skip-inject.""" + items: list[str] = [] + for n in notes: + text = (n.get("text") or "").strip() + if not text: + continue + permalink = n.get("permalink") + items.append(f"{text}\n— [[{permalink}]]" if permalink else text) + if not items: + return "" + return "## Relevant memories (agentcairn)\n\n" + "\n\n---\n\n".join(items) + + +def build_hook_output(block: str) -> dict: + """Wrap an injection block in the Claude Code UserPromptSubmit envelope.""" + return { + "hookSpecificOutput": { + "hookEventName": "UserPromptSubmit", + "additionalContext": block, + } + } + + +def _recall(prompt: str, *, vault, index, embedder_name: str, k: int, scope: str) -> list[dict]: + """Run one hybrid recall; returns note dicts (possibly empty). Falls back to + BM25-only if the embedder cannot load. Records the savings ledger + best-effort (this is the observability that proves recall fired).""" + idx = paths.index_for(index, paths.resolve_vault(vault)) + if not idx.exists(): + return [] + try: + emb = None if embedder_name == "none" else get_embedder(embedder_name) + except Exception: + emb = None # BM25-only fallback when the embedder can't load + current = resolve_current_project(None) + con = open_search(str(idx)) + try: + hits = search(con, prompt, embedder=emb, k=k, rerank=False, project=current, scope=scope) + notes = [ + {"permalink": h.permalink, "title": h.heading_path, "text": h.snippet, "score": h.score} + for h in hits + ] + try: + from cairn import usage + from cairn.index.schema import cached_haystack_tokens + + full = cached_haystack_tokens(con) + recalled = sum(usage.estimate_tokens(n["text"]) for n in notes) + usage.record("recall", full=full, recalled=recalled, k=k) + except Exception: + pass + finally: + con.close() + return notes + + +def run( + stdin_text: str, + *, + vault: Path | str | None = None, + index: Path | str | None = None, + embedder_name: str = "fastembed", + env: Mapping[str, str] | None = None, +) -> str: + """Parse a UserPromptSubmit payload (JSON on stdin) and return the string to + print: a hook-output JSON envelope, or "" to inject nothing. NEVER raises — + every failure path returns "" (fail-open).""" + try: + if env is None: + env = cairn_env() + try: + prompt = (json.loads(stdin_text) or {}).get("prompt") or "" + except (ValueError, TypeError): + prompt = "" + if not should_recall(prompt, env): + return "" + notes = _recall( + prompt, + vault=vault, + index=index, + embedder_name=embedder_name, + k=resolve_auto_recall_k(env), + scope=resolve_auto_recall_scope(env), + ) + block = format_block(notes) + return json.dumps(build_hook_output(block)) if block else "" + except Exception: + return "" +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run pytest tests/test_recall_hook.py -v` +Expected: PASS (9 passed). If `test_run_injects_relevant_memory` finds no hits, confirm `build_index` is imported from `tests.search.test_engine` and the prompt overlaps the fixture's "coffee"/"beans" note. + +- [ ] **Step 5: Commit** + +```bash +git add src/cairn/recall_hook.py tests/test_recall_hook.py +git commit # message: "feat(recall): recall_hook core (gate, recall, format, fail-open)" + trailer +``` + +--- + +### Task 3: `cairn recall-hook` CLI command + +**Files:** +- Modify: `src/cairn/cli.py` (add a command; mirror the `recall` command's option style ~line 280) +- Test: `tests/test_recall_hook_cli.py` (create) + +**Interfaces:** +- Consumes (Task 2): `cairn.recall_hook.run`. +- Produces: CLI command `cairn recall-hook [--vault PATH] [--index PATH] [--embedder NAME]` that reads the hook JSON from stdin and prints `run(...)`'s output (or nothing). Always exits 0. + +- [ ] **Step 1: Write the failing test** + +Create `tests/test_recall_hook_cli.py`: + +```python +# SPDX-License-Identifier: Apache-2.0 +import json + +from typer.testing import CliRunner + +from cairn.cli import app +from cairn.embed import get_embedder +from tests.search.test_engine import build_index + +runner = CliRunner() + + +def test_recall_hook_cli_stdin_wiring(tmp_path): + idx = build_index(tmp_path, get_embedder("fake")) + r = runner.invoke( + app, + ["recall-hook", "--index", str(idx), "--embedder", "fake"], + input=json.dumps({"prompt": "how do I brew coffee beans?"}), + ) + assert r.exit_code == 0 + data = json.loads(r.output) + assert data["hookSpecificOutput"]["hookEventName"] == "UserPromptSubmit" + + +def test_recall_hook_cli_trivial_prompt_no_output(tmp_path): + idx = build_index(tmp_path, get_embedder("fake")) + r = runner.invoke( + app, + ["recall-hook", "--index", str(idx), "--embedder", "fake"], + input=json.dumps({"prompt": "go"}), + ) + assert r.exit_code == 0 + assert r.output.strip() == "" +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `uv run pytest tests/test_recall_hook_cli.py -v` +Expected: FAIL — `recall-hook` is not a known command (Typer exits non-zero / "No such command"). + +- [ ] **Step 3: Add the command** + +In `src/cairn/cli.py`, add (near the `recall` command; `Path`, `typer`, and `app` are already imported there): + +```python +@app.command("recall-hook") +def recall_hook( + vault: Path = typer.Option( + None, "--vault", help="Vault dir (default: CAIRN_VAULT or ~/agentcairn)." + ), + index: Path = typer.Option( + None, "--index", help="Index .duckdb path (default: derived from vault)." + ), + embedder: str = typer.Option( + "fastembed", "--embedder", help="'fastembed' (default), 'fake' (tests), or 'none' (BM25)." + ), +) -> None: + """Auto-recall for the Claude Code UserPromptSubmit hook (internal). + + Reads the hook JSON payload from stdin, runs a hybrid recall against the + prompt, and prints the additionalContext envelope (or nothing). Always + exits 0 — never blocks or breaks a prompt. + """ + import sys + + from cairn import recall_hook as _rh + + out = _rh.run(sys.stdin.read(), vault=vault, index=index, embedder_name=embedder) + if out: + typer.echo(out) +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `uv run pytest tests/test_recall_hook_cli.py -v` +Expected: PASS (2 passed). + +- [ ] **Step 5: Run the full suite** + +Run: `uv run pytest -q` +Expected: PASS (no regressions). + +- [ ] **Step 6: Commit** + +```bash +git add src/cairn/cli.py tests/test_recall_hook_cli.py +git commit # message: "feat(cli): cairn recall-hook command (stdin -> recall_hook.run)" + trailer +``` + +--- + +### Task 4: Plugin wiring (hook + wrapper + warm + version) + +**Files:** +- Create: `plugin/scripts/user-prompt-submit.sh` (chmod 0755) +- Modify: `plugin/hooks/hooks.json` (add `UserPromptSubmit`) +- Modify: `plugin/scripts/session-start.sh` (add detached warm on the warm path) +- Modify: `plugin/.claude-plugin/plugin.json` (`0.3.1` → `0.4.0`) +- Test: `plugin/tests/test_plugin.py` (add a wiring assertion) + +**Interfaces:** +- Consumes (Task 3): the `cairn recall-hook` command. +- The wrapper is **synchronous** — its stdout is the injected context. It is NOT a detached `cairn warm`; warm-keeping lives in `session-start.sh`. + +- [ ] **Step 1: Write the failing plugin-wiring test** + +In `plugin/tests/test_plugin.py`, add (mirrors the existing `test_precompact_hook_*`; `_json`, `PLUGIN` already defined at module top): + +```python +def test_userpromptsubmit_hook_runs_recall(): + hooks = _json(PLUGIN / "hooks" / "hooks.json")["hooks"] + assert "UserPromptSubmit" in hooks + assert "user-prompt-submit.sh" in json.dumps(hooks["UserPromptSubmit"]) +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `uv run pytest plugin/tests/test_plugin.py::test_userpromptsubmit_hook_runs_recall -v` +Expected: FAIL — `"UserPromptSubmit"` not in `hooks`. + +- [ ] **Step 3: Add the `UserPromptSubmit` entry to `hooks.json`** + +Replace the contents of `plugin/hooks/hooks.json` with (adds `UserPromptSubmit` first; mind the trailing comma rules): + +```json +{ + "hooks": { + "UserPromptSubmit": [ + { "matcher": "*", "hooks": [ + { "type": "command", "command": "sh", + "args": ["${CLAUDE_PLUGIN_ROOT}/scripts/user-prompt-submit.sh"], + "timeout": 10 } ] } + ], + "SessionStart": [ + { "matcher": "*", "hooks": [ + { "type": "command", "command": "sh", + "args": ["${CLAUDE_PLUGIN_ROOT}/scripts/session-start.sh"], + "timeout": 20 } ] } + ], + "SessionEnd": [ + { "matcher": "*", "hooks": [ + { "type": "command", "command": "sh", + "args": ["${CLAUDE_PLUGIN_ROOT}/scripts/session-end.sh"], + "timeout": 120 } ] } + ], + "PreCompact": [ + { "matcher": "*", "hooks": [ + { "type": "command", "command": "sh", + "args": ["${CLAUDE_PLUGIN_ROOT}/scripts/session-end.sh"], + "timeout": 30 } ] } + ] + } +} +``` + +- [ ] **Step 4: Create the wrapper script** + +Create `plugin/scripts/user-prompt-submit.sh`: + +```sh +#!/bin/sh +# UserPromptSubmit hook. Runs a hybrid recall against the user's prompt and +# prints it as additionalContext for this turn. SYNCHRONOUS — its stdout IS the +# injected context (do not detach it). Fail-open: `cairn recall-hook` always +# exits 0 and emits nothing on any problem, so it never blocks or breaks a +# prompt. The 10s hook timeout is the safety ceiling; SessionStart pre-warms the +# embedder so the steady-state path is ~1s. stdin = the UserPromptSubmit hook +# JSON (the prompt), inherited by the command. +set -u +VAULT=$(printf '%s' "${CLAUDE_PLUGIN_OPTION_VAULT_PATH:-${1:-$HOME/agentcairn}}" | sed "s#^~#$HOME#") +CAIRN="uvx --from agentcairn>=0.2 cairn" +$CAIRN recall-hook --vault "$VAULT" 2>/dev/null +exit 0 +``` + +Then make it executable: + +```bash +chmod 0755 plugin/scripts/user-prompt-submit.sh +``` + +- [ ] **Step 5: Add detached warm to `session-start.sh`** + +In `plugin/scripts/session-start.sh`, immediately AFTER the first-run `if … fi` block (the block that runs `( $CAIRN init "$VAULT"; $CAIRN warm ) … &` then `exit 0`) and BEFORE the `# Fetch recent memories as JSON` comment, insert: + +```sh +# Keep the embedder/reranker models loaded on every (warm-path) session so the +# per-prompt UserPromptSubmit recall stays fast. `cairn warm` is idempotent and +# near-instant once cached; fully detached anyway so a cold re-download can never +# delay the session, and stdin/stdout/stderr detached so it can't hold the hook's +# pipes open. Best-effort: failures are swallowed. +( $CAIRN warm ) /dev/null 2>&1 & +``` + +- [ ] **Step 6: Bump the plugin version** + +In `plugin/.claude-plugin/plugin.json`, change `"version": "0.3.1"` to `"version": "0.4.0"`. + +- [ ] **Step 7: Run the plugin tests (full file — outside default testpaths)** + +Run: `uv run pytest plugin/tests/test_plugin.py -v` +Expected: PASS — including the new `test_userpromptsubmit_hook_runs_recall` AND the existing `test_hooks_do_not_hardfail_on_unset_vault_path` (the new entry uses `${CLAUDE_PLUGIN_ROOT}`, no `user_config`). + +- [ ] **Step 8: Commit** + +```bash +git add plugin/hooks/hooks.json plugin/scripts/user-prompt-submit.sh \ + plugin/scripts/session-start.sh plugin/.claude-plugin/plugin.json \ + plugin/tests/test_plugin.py +git commit # message: "feat(plugin): UserPromptSubmit auto-recall hook + warm-path warm (v0.4.0)" + trailer +``` + +--- + +### Task 5: Docs + CHANGELOG + +**Files:** +- Modify: `README.md` (document auto-recall + the three config keys) +- Modify: `CHANGELOG.md` (Unreleased entry) + +**Interfaces:** +- Consumes: nothing (documentation of Tasks 1-4). + +- [ ] **Step 1: Document auto-recall in the README** + +In `README.md`, find the section describing the Claude Code plugin's ambient behavior (recall-at-start + capture-at-end). Add a short paragraph and a config block. Use this copy: + +```markdown +**Automatic recall (Claude Code).** On every substantive prompt, the plugin runs +a hybrid recall against what you just asked and injects the most relevant +memories as context for that turn — not just the recency digest shown at session +start. Trivially-short prompts (e.g. "yes", "go") are skipped. It is fail-open: +if anything goes wrong it injects nothing and never blocks your prompt. + +Configure it in `~/.agentcairn/config.toml` (flat top-level keys; env vars +`CAIRN_AUTO_RECALL{,_K,_SCOPE}` override the file): + +​```toml +auto_recall = true # master on/off (default: true) +auto_recall_k = 3 # memories injected per prompt +auto_recall_scope = "all" # "all" (boost, non-lossy) or "project" (hard filter) +​``` +``` + +(Remove the zero-width `​` characters around the inner fence — they are only here to nest the code block in this plan. The README gets a normal ```` ```toml ```` block.) + +- [ ] **Step 2: Add a CHANGELOG entry** + +In `CHANGELOG.md`, under the `## [Unreleased]` heading (create it above the latest version if absent), add: + +```markdown +### Added +- **Automatic per-turn recall on Claude Code.** A `UserPromptSubmit` hook runs a + hybrid recall against each substantive prompt and injects the hits as context, + closing the gap where the `recall` tool was almost never invoked. New + `cairn recall-hook` command; configurable via `auto_recall` / `auto_recall_k` / + `auto_recall_scope` (default on). Plugin `0.4.0`. +``` + +- [ ] **Step 3: Verify the suite is still green** + +Run: `uv run pytest -q` +Expected: PASS. + +- [ ] **Step 4: Commit** + +```bash +git add README.md CHANGELOG.md +git commit # message: "docs: document automatic per-turn recall + config" + trailer +``` + +--- + +## Self-Review + +**Spec coverage:** +- Behavior (substantive-prompt recall + inject) → Tasks 2, 3, 4. ✓ +- Keep SessionStart digest → unchanged (Task 4 only *adds*). ✓ +- Default-on, configurable (`auto`/`k`/`scope`, flat keys) → Task 1 + README (Task 5). ✓ +- Trivial-prompt gate → `should_recall` (Task 2). ✓ +- Architecture: thin command + shell wrapper → Tasks 2-4. ✓ +- Latency: SessionStart pre-warm + BM25 fallback + 10s ceiling → Task 4 warm line, `_recall` except→`None`, `hooks.json` timeout. ✓ +- Fail-open (always exit 0) → `run()` try/except, wrapper `exit 0`, command no-raise. ✓ +- Injection format (`## Relevant memories (agentcairn)` + permalink) → `format_block` (Task 2). ✓ +- usage.jsonl moves again → `usage.record` in `_recall` (Task 2). ✓ +- Testing (unit + CliRunner stdin + plugin wiring) → Tasks 1-4. ✓ +- Codex fast-follow → out of scope (noted). ✓ + +**Placeholder scan:** none — every code step shows complete code. The README inner-fence note explains the zero-width placeholder characters. + +**Type consistency:** `run()` signature is identical in Task 2 (definition) and Task 3 (call). `Hit` attributes used (`.permalink`, `.heading_path`, `.snippet`, `.score`) match the extracted dataclass. `format_block` consumes the dict keys (`permalink`, `text`) that `_recall` produces. Resolver names match between Task 1 (defs) and Task 2 (imports). diff --git a/docs/specs/2026-06-29-auto-recall-userpromptsubmit-design.md b/docs/specs/2026-06-29-auto-recall-userpromptsubmit-design.md new file mode 100644 index 0000000..e6db614 --- /dev/null +++ b/docs/specs/2026-06-29-auto-recall-userpromptsubmit-design.md @@ -0,0 +1,233 @@ +# Automatic per-turn recall (UserPromptSubmit) — design + +**Date:** 2026-06-29 +**Status:** approved (brainstorm); pending implementation plan +**Scope:** Claude Code plugin (Codex is a verified fast-follow, out of scope here) + +## Problem + +agentcairn is currently **write-mostly** on Claude Code: sessions are captured and +the index is healthy, but *semantic recall is essentially never triggered*. + +Empirical evidence (user's machine, 2026-06-29): + +- `~/.cache/agentcairn/usage.jsonl` records **only** `recall` events. The last + entry was `2026-06-24T18:38` — **zero recalls in ~4 days of active multi-session + use**, despite many session starts. +- `cairn recall` works correctly (returns ranked, cited results); the index is + healthy (696 notes / 892 chunks / 892 embeds, local `nomic-embed-text-v1.5`). +- So the gap is on the **trigger** side, not capture, indexing, or the engine. + +Root cause, confirmed in code: + +- The Claude Code plugin wires only `SessionStart`, `SessionEnd`, `PreCompact` + hooks — **no per-prompt hook**. `SessionStart` injects `cairn recent -n 5`, a + pure `ORDER BY mtime DESC` **recency digest** (no query, no embeddings, no + ranking) — and `recent` does **not** write to `usage.jsonl`, which is exactly + why the ledger stayed frozen. +- The `recall` MCP tool and the `using-agentcairn-memory` skill exist but are + **LLM-discretion-gated** — they fire only if the model chooses to invoke them, + and in practice it does not. +- Per-prompt auto-recall was **deliberately omitted** in the original plugin spec + (`docs/specs/2026-06-10-agentcairn-claude-code-plugin-design.md` §13) for + "latency/noise/cost" reasons. This design revisits that decision: the empirical + cost of the omission is that recall never happens at all. + +Notably, agentcairn's **OpenCode** and **Hermes** integrations already do +automatic per-turn relevance recall (OpenCode via `chat.message` + +`experimental.chat.system.transform → cairn recall`; Hermes via `prefetch()`). +The flagship host is the laggard. This design ports that proven pattern to Claude +Code's `UserPromptSubmit` hook. + +## Goals + +- On each substantive Claude Code user prompt, run a hybrid recall against the + prompt and inject the top hits as context for that turn. +- Keep the existing `SessionStart` recency digest unchanged (complementary: + recency-orientation at start + relevance per turn). +- Default-on, configurable via `~/.agentcairn/config.toml`. +- Never block, slow, or break a prompt (fail-open). + +## Non-goals (YAGNI) + +- Codex port (verified fast-follow once its per-prompt hook capability is + confirmed). +- Topic-aware / "new topic" gating (a simple length gate suffices). +- Deduplication of per-turn recall against the `SessionStart` digest (recency vs + relevance overlap is minimal; dedup adds hot-path latency). +- A resident embedder daemon (pre-warm + per-call load is sufficient). + +## Settled decisions (from brainstorm) + +1. **Keep both** — `SessionStart` digest *and* per-turn relevance recall, + complementary. +2. **Claude Code only** now; Codex fast-follow. +3. **Skip trivially-short prompts** via a simple length gate. +4. **Default-on, configurable** in `config.toml` (`auto`, `k`, `scope`). +5. **Architecture:** a thin Python command `cairn recall-hook` holds all logic; a + small shell wrapper wires it to the hook (mirrors the project's "thin shell + over the cairn CLI" philosophy, e.g. the OpenCode TS plugin). + +## Architecture + +### New files + +- **`src/cairn/recall_hook.py`** — all logic, as pure/testable units: + - `should_recall(prompt: str, cfg) -> bool` — config master-switch **and** the + trivial-prompt length gate. + - `format_block(notes: list[dict]) -> str` — render recalled notes into the + injection markdown; returns `""` when there is nothing to inject. + - `build_hook_output(block: str) -> dict` — the Claude Code `UserPromptSubmit` + hook JSON envelope. + - `run(stdin_text: str, env, vault) -> str` — orchestrates: parse stdin → + `should_recall` → hybrid `search()` → `format_block` → `build_hook_output`; + returns the string to print (empty string ⇒ print nothing). +- **`plugin/scripts/user-prompt-submit.sh`** — thin wrapper. Resolves the vault + exactly like `session-start.sh` + (`${CLAUDE_PLUGIN_OPTION_VAULT_PATH:-${1:-$HOME/agentcairn}}`, `~`-expanded), + then `exec`s `cairn recall-hook --vault "$VAULT"` with stdin inherited. +- **`tests/test_recall_hook.py`** — unit tests (see Testing). + +### Modified files + +- **`src/cairn/cli.py`** — add `@app.command("recall-hook")` that reads stdin and + delegates to `recall_hook.run(...)`, printing its result. Reuses the existing + `search()` / embedder-resolution path shared with `recall`. +- **`plugin/hooks/hooks.json`** — add a `UserPromptSubmit` entry calling + `user-prompt-submit.sh` with a safety-ceiling `timeout` (~10s; the BM25 + cold-fallback keeps the expected latency far below this, and on timeout Claude + Code proceeds without injection — fail-open). +- **`plugin/scripts/session-start.sh`** — fire a detached, best-effort + `cairn warm` on each start to pre-load the embedder (idempotent; cheap when the + model is already cached). The first-run path already does this; generalize it. +- **`src/cairn/config.py`** — add `KNOBS` entries + `resolve_*()` functions for + the recall knobs (see Config). +- **`README.md`** + docs — document automatic recall and the `[recall]` config. + +## Config surface + +**Flat top-level keys** in `~/.agentcairn/config.toml`. The config loader +(`config_file_values`) reads only top-level keys — it does **not** descend into +`[section]` tables, and every existing knob (`vault`, `rerank`, `usage`, …) is +flat. Each key maps to a `CAIRN_*` env var through the existing `KNOBS` / +`_translate` machinery (`foo` → `CAIRN_FOO`), preserving the established +precedence: **explicit arg → env → config file → default**. + +```toml +auto_recall = true # CAIRN_AUTO_RECALL — master on/off (default: true) +auto_recall_k = 3 # CAIRN_AUTO_RECALL_K — notes injected per turn (default: 3) +auto_recall_scope = "all" # CAIRN_AUTO_RECALL_SCOPE — "all" (boost, non-lossy) | "project" (hard filter) +``` + +- `resolve_auto_recall(env) -> bool` (default `True`) +- `resolve_auto_recall_k(env) -> int` (default `3`) +- `resolve_auto_recall_scope(env) -> str` (default `"all"`) + +Rerank is **forced off** on this hot path (latency). The trivial-prompt length +threshold is a module constant (env-overridable via `CAIRN_AUTO_RECALL_MIN_CHARS`, +default ~12) but intentionally **not** surfaced in `config.toml` (YAGNI). + +## Data flow (per turn) + +1. User submits a prompt. Claude Code fires the `UserPromptSubmit` hook, invoking + `user-prompt-submit.sh` with the hook JSON on **stdin** + (`{prompt, cwd, session_id, …}`). +2. Wrapper resolves the vault and `exec`s `cairn recall-hook --vault "$VAULT"`, + stdin inherited. +3. `recall-hook` parses stdin and extracts `prompt`. If `auto` is disabled **or** + `should_recall` rejects the prompt (too short) → **exit 0, no output**. +4. Otherwise it runs the hybrid `search()` (`k`, `scope`, `project=cwd` boost, + **no rerank**), formats the block, and prints: + ```json + {"hookSpecificOutput":{"hookEventName":"UserPromptSubmit", + "additionalContext":"## Relevant memories (agentcairn)\n\n…"}} + ``` + then exits 0. +5. Claude Code injects `additionalContext` into the model's context for that turn. +6. The recall path calls `usage.record("recall", …)`, so `usage.jsonl` starts + moving again and the savings stat reflects **real** interactive recall. + +### Injection format + +Mirrors the OpenCode plugin's `formatMemoryBlock` for cross-host consistency, with +a permalink so the model can cite (per the `using-agentcairn-memory` skill): + +``` +## Relevant memories (agentcairn) + + +— [[]] + +--- + + +— [[]] +``` + +Recall JSON shape consumed (from `cairn recall --json`): +`[{permalink, title, text, score}, …]`. + +## The trivial-prompt gate + +`should_recall` skips recall when `len(prompt.strip())` is below the threshold +(~12 chars). Drops bare continuations ("yes", "go", "A"); keeps substantive short +prompts ("lets check on the pr's"). No topic detection. + +## Latency / warm strategy + +- **Pre-warm:** `SessionStart` spawns a detached `cairn warm` so the embedder's + model files are cached on disk. +- **Per-call:** each `recall-hook` invocation loads the model from cache + (~0.5–1.5s warm). OpenCode already shells `cairn recall` per turn, so this is + proven tolerable in practice. +- **Fail-open + BM25 fallback:** if the embedder cannot load, `recall-hook` + degrades to **BM25-only** (`embedder=None`) for that turn; on any other + error/timeout it injects nothing. The genuine cold case (first-ever run) has no + index yet, so the hook exits early regardless. +- The hook `timeout` (~10s) is the hard ceiling: if a cold model load exceeds it, + Claude Code kills the hook and the prompt proceeds without injection + (fail-open). An active embedder-warmth check (e.g. a thread deadline) is + deferred — the blocking path is rarely hit in practice. + +## Error handling — fail-open, always + +A `UserPromptSubmit` hook that exits non-zero **blocks the prompt**. Therefore +`recall-hook` **always exits 0** and prints nothing on any problem: missing index, +stdin parse error, recall exception, timeout, or empty results. Every path is +wrapped. Recalled content is already redacted at write time (redaction-before- +write), so the hook needs no additional redaction. + +## Testing + +**Unit (`tests/test_recall_hook.py`):** + +- `should_recall`: trivial prompts skipped; substantive prompts pass; respects + `auto=false`. +- Config resolution: `resolve_auto_recall{,_k,_scope}` honor env and `config.toml` + with correct precedence and default-on. +- `format_block`: empty/all-blank notes → `""` (no injection); populated → correct + markdown with permalinks. +- `build_hook_output`: correct `UserPromptSubmit` envelope. +- Fail-open: no index / malformed stdin → `run` returns `""` and the command exits + 0 (never raises). +- BM25 cold-fallback: when the embedder is unavailable, recall still returns + keyword hits rather than erroring. + +**Plugin wiring (`plugin/tests/`):** assert `hooks.json` contains a +`UserPromptSubmit` entry pointing at `user-prompt-submit.sh`. (Note: `plugin/tests/` +runs only under CI's `validate` job or manually — `uv run pytest plugin/tests/` — +not the default `testpaths`.) + +## Cross-host note + +OpenCode and Hermes already auto-recall per turn. After this lands, Claude Code +reaches parity. The **Codex** plugin shares `session-start.sh` and would take the +same wrapper + `hooks.codex.json` `UserPromptSubmit`-equivalent entry **iff** Codex +exposes a per-prompt hook — to be verified, then shipped as a fast-follow. + +## Rollout + +- Plugin version bump (`plugin/.claude-plugin/plugin.json`). +- CHANGELOG entry. +- No index/schema migration; no breaking change. Existing users get auto-recall on + next plugin update (default-on); opt out via `config.toml`/`CAIRN_AUTO_RECALL=0`. diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index 6b28727..e1de30f 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "agentcairn", "displayName": "agentcairn", "description": "Local-first agent memory for Claude Code — recall, remember, and ambient capture into a Markdown vault you own.", - "version": "0.3.1", + "version": "0.4.0", "author": { "name": "Charles C. Figueiredo", "email": "ccf@ccf.io" }, "homepage": "https://agentcairn.dev", "repository": "https://github.com/ccf/agentcairn", diff --git a/plugin/hooks/hooks.json b/plugin/hooks/hooks.json index 83e9fac..f6aefd1 100644 --- a/plugin/hooks/hooks.json +++ b/plugin/hooks/hooks.json @@ -1,5 +1,11 @@ { "hooks": { + "UserPromptSubmit": [ + { "matcher": "*", "hooks": [ + { "type": "command", "command": "sh", + "args": ["${CLAUDE_PLUGIN_ROOT}/scripts/user-prompt-submit.sh"], + "timeout": 10 } ] } + ], "SessionStart": [ { "matcher": "*", "hooks": [ { "type": "command", "command": "sh", diff --git a/plugin/scripts/session-start.sh b/plugin/scripts/session-start.sh index 8619152..24afd7c 100755 --- a/plugin/scripts/session-start.sh +++ b/plugin/scripts/session-start.sh @@ -33,6 +33,13 @@ if [ ! -d "$HOME/.cache/agentcairn/indexes" ] && [ ! -f "$HOME/.cache/agentcairn exit 0 fi +# Keep the embedder/reranker models loaded on every (warm-path) session so the +# per-prompt UserPromptSubmit recall stays fast. `cairn warm` is idempotent and +# near-instant once cached; fully detached anyway so a cold re-download can never +# delay the session, and stdin/stdout/stderr detached so it can't hold the hook's +# pipes open. Best-effort: failures are swallowed. +( $CAIRN warm ) /dev/null 2>&1 & + # Fetch recent memories as JSON (best-effort, cross-project). JSON=$($CAIRN recent --vault "$VAULT" -n 5 --json 2>/dev/null || echo '{"notes":[]}') diff --git a/plugin/scripts/user-prompt-submit.sh b/plugin/scripts/user-prompt-submit.sh new file mode 100755 index 0000000..b1364fd --- /dev/null +++ b/plugin/scripts/user-prompt-submit.sh @@ -0,0 +1,13 @@ +#!/bin/sh +# UserPromptSubmit hook. Runs a hybrid recall against the user's prompt and +# prints it as additionalContext for this turn. SYNCHRONOUS — its stdout IS the +# injected context (do not detach it). Fail-open: `cairn recall-hook` always +# exits 0 and emits nothing on any problem, so it never blocks or breaks a +# prompt. The 10s hook timeout is the safety ceiling; SessionStart pre-warms the +# embedder so the steady-state path is ~1s. stdin = the UserPromptSubmit hook +# JSON (the prompt), inherited by the command. +set -u +VAULT=$(printf '%s' "${CLAUDE_PLUGIN_OPTION_VAULT_PATH:-${1:-$HOME/agentcairn}}" | sed "s#^~#$HOME#") +CAIRN="uvx --from agentcairn>=0.2 cairn" +$CAIRN recall-hook --vault "$VAULT" 2>/dev/null +exit 0 diff --git a/plugin/tests/test_plugin.py b/plugin/tests/test_plugin.py index 565fff0..53d0e34 100644 --- a/plugin/tests/test_plugin.py +++ b/plugin/tests/test_plugin.py @@ -267,6 +267,12 @@ def test_hooks_do_not_hardfail_on_unset_vault_path(): assert "session-start.sh" in blob and "session-end.sh" in blob +def test_userpromptsubmit_hook_runs_recall(): + hooks = _json(PLUGIN / "hooks" / "hooks.json")["hooks"] + assert "UserPromptSubmit" in hooks + assert "user-prompt-submit.sh" in json.dumps(hooks["UserPromptSubmit"]) + + def test_precompact_hook_captures_long_sessions(): # Capture must not wait for SessionEnd: long/resumed sessions compact # repeatedly, so PreCompact runs the same detached sweep at each boundary, diff --git a/src/cairn/cli.py b/src/cairn/cli.py index 8fe89d7..5c5b403 100644 --- a/src/cairn/cli.py +++ b/src/cairn/cli.py @@ -362,6 +362,36 @@ def recall( typer.echo(f" {h.snippet.strip()[:160]}") +@app.command("recall-hook") +def recall_hook( + vault: Path = typer.Option( + None, "--vault", help="Vault dir (default: CAIRN_VAULT or ~/agentcairn)." + ), + index: Path = typer.Option( + None, "--index", help="Index .duckdb path (default: derived from vault)." + ), + embedder: str = typer.Option( + None, "--embedder", help="'fastembed' (default), 'fake' (tests), or 'none' (BM25)." + ), +) -> None: + """Auto-recall for the Claude Code UserPromptSubmit hook (internal). + + Reads the hook JSON payload from stdin, runs a hybrid recall against the + prompt, and prints the additionalContext envelope (or nothing). Always + exits 0 — never blocks or breaks a prompt. + """ + try: + import sys + + from cairn import recall_hook as _rh + + out = _rh.run(sys.stdin.read(), vault=vault, index=index, embedder_name=embedder) + if out: + typer.echo(out) + except Exception: + pass + + @app.command() def recent( index: Path = typer.Option(None, "--index", help="Index .duckdb path."), diff --git a/src/cairn/config.py b/src/cairn/config.py index d00a34c..e808fea 100644 --- a/src/cairn/config.py +++ b/src/cairn/config.py @@ -101,6 +101,24 @@ class Knob: "true", "Semantic dedup + supersession during ingest (LLM judge tier only).", ), + Knob( + "auto_recall", + "CAIRN_AUTO_RECALL", + "true", + "Auto-recall relevant memory before each substantive prompt (Claude Code).", + ), + Knob( + "auto_recall_k", + "CAIRN_AUTO_RECALL_K", + "3", + "How many memories auto-recall injects per prompt.", + ), + Knob( + "auto_recall_scope", + "CAIRN_AUTO_RECALL_SCOPE", + "all", + "Auto-recall scope: 'all' (boost, non-lossy) or 'project' (hard filter).", + ), ) _KNOWN_KEYS = {k.key for k in KNOBS} @@ -209,6 +227,9 @@ def openai_config(env: Mapping[str, str] | None = None) -> tuple[str, str | None return model, env.get("OPENAI_API_KEY"), base +_DEFAULT_AUTO_RECALL_K = 3 + + def resolve_rerank(explicit: bool | None = None, env: Mapping[str, str] | None = None) -> bool: """Resolve the reranker on/off setting: explicit arg → CAIRN_RERANK env → True. An unparseable CAIRN_RERANK falls back to the default (True) rather than raising, @@ -241,6 +262,39 @@ def resolve_consolidate(env: Mapping[str, str] | None = None) -> bool: return True +def resolve_auto_recall(env: Mapping[str, str] | None = None) -> bool: + """Resolve auto-recall on/off: CAIRN_AUTO_RECALL env/file → True. + An unparseable value falls back to the default (True) rather than raising, + so a typo never disables recall silently or breaks a prompt.""" + if env is None: + env = cairn_env() + raw = env.get("CAIRN_AUTO_RECALL") + if raw is None: + return True + try: + return parse_bool(raw) + except ValueError: + return True + + +def resolve_auto_recall_k(env: Mapping[str, str] | None = None) -> int: + """Resolve auto-recall depth: CAIRN_AUTO_RECALL_K env/file → 3. + An unparseable value falls back to the default rather than raising.""" + if env is None: + env = cairn_env() + try: + return int(env.get("CAIRN_AUTO_RECALL_K") or _DEFAULT_AUTO_RECALL_K) + except ValueError: + return _DEFAULT_AUTO_RECALL_K + + +def resolve_auto_recall_scope(env: Mapping[str, str] | None = None) -> str: + """Resolve auto-recall scope: CAIRN_AUTO_RECALL_SCOPE env/file → 'all'.""" + if env is None: + env = cairn_env() + return (env.get("CAIRN_AUTO_RECALL_SCOPE") or "all").strip().lower() + + _DEFAULT_JUDGE_MODEL = "claude-haiku-4-5" _DEFAULT_JUDGE_TIMEOUT = 90.0 # floor; LLMJudge scales the real budget per batch diff --git a/src/cairn/recall_hook.py b/src/cairn/recall_hook.py new file mode 100644 index 0000000..0946d09 --- /dev/null +++ b/src/cairn/recall_hook.py @@ -0,0 +1,150 @@ +# SPDX-License-Identifier: Apache-2.0 +"""UserPromptSubmit auto-recall. + +Runs a hybrid recall against the user's prompt and emits it as Claude Code +`additionalContext`. All logic lives here as small, testable units; the plugin +ships only a thin shell wrapper that execs the `cairn recall-hook` CLI command, +which delegates to `run()`. Every path is fail-open: `run()` never raises and +returns "" (inject nothing) on any problem.""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from pathlib import Path + +from cairn import paths +from cairn.config import ( + cairn_env, + resolve_auto_recall, + resolve_auto_recall_k, + resolve_auto_recall_scope, +) +from cairn.embed import get_embedder +from cairn.ingest.events import project_from_cwd +from cairn.search import open_search, resolve_current_project, search + +_DEFAULT_MIN_CHARS = 12 + + +def _min_chars(env: Mapping[str, str]) -> int: + try: + return int(env.get("CAIRN_AUTO_RECALL_MIN_CHARS") or _DEFAULT_MIN_CHARS) + except ValueError: + return _DEFAULT_MIN_CHARS + + +def should_recall(prompt: str, env: Mapping[str, str] | None = None) -> bool: + """True iff auto-recall is enabled and the prompt is substantive. + Skips trivially-short prompts ("yes", "go") — continuations where recall + adds noise, not signal.""" + if env is None: + env = cairn_env() + if not resolve_auto_recall(env): + return False + return len(prompt.strip()) >= _min_chars(env) + + +def format_block(notes: list[dict]) -> str: + """Render recalled notes into the injection markdown. Returns "" when there + is nothing to inject (empty list / all-blank texts) so callers skip-inject.""" + items: list[str] = [] + for n in notes: + text = (n.get("text") or "").strip() + if not text: + continue + permalink = n.get("permalink") + items.append(f"{text}\n— [[{permalink}]]" if permalink else text) + if not items: + return "" + return "## Relevant memories (agentcairn)\n\n" + "\n\n---\n\n".join(items) + + +def build_hook_output(block: str) -> dict: + """Wrap an injection block in the Claude Code UserPromptSubmit envelope.""" + return { + "hookSpecificOutput": { + "hookEventName": "UserPromptSubmit", + "additionalContext": block, + } + } + + +def _recall( + prompt: str, *, vault, index, embedder_name: str, k: int, scope: str, cwd: str | None = None +) -> list[dict]: + """Run one hybrid recall; returns note dicts (possibly empty). Falls back to + BM25-only if the embedder cannot load. Records the savings ledger + best-effort (this is the observability that proves recall fired).""" + idx = paths.index_for(index, paths.resolve_vault(vault)) + if not idx.exists(): + return [] + try: + emb = None if embedder_name == "none" else get_embedder(embedder_name) + except Exception: + emb = None # BM25-only fallback when the embedder can't load + # Project boost/scope resolves from the hook payload's cwd (the session + # workspace), falling back to the process cwd when the payload omits it. + current = resolve_current_project( + project_from_cwd(cwd) if isinstance(cwd, str) and cwd else None + ) + notes: list[dict] = [] + con = open_search(str(idx)) + try: + hits = search(con, prompt, embedder=emb, k=k, rerank=False, project=current, scope=scope) + notes = [ + {"permalink": h.permalink, "title": h.heading_path, "text": h.snippet, "score": h.score} + for h in hits + ] + try: + from cairn import usage + from cairn.index.schema import cached_haystack_tokens + + full = cached_haystack_tokens(con) + recalled = sum(usage.estimate_tokens(n["text"]) for n in notes) + usage.record("recall", full=full, recalled=recalled, k=k) + except Exception: + pass + finally: + con.close() + return notes + + +def run( + stdin_text: str, + *, + vault: Path | str | None = None, + index: Path | str | None = None, + embedder_name: str | None = None, + env: Mapping[str, str] | None = None, +) -> str: + """Parse a UserPromptSubmit payload (JSON on stdin) and return the string to + print: a hook-output JSON envelope, or "" to inject nothing. NEVER raises — + every failure path returns "" (fail-open).""" + try: + if env is None: + env = cairn_env() + name = embedder_name or env.get("CAIRN_EMBEDDER") or "fastembed" + try: + obj = json.loads(stdin_text) + except (ValueError, TypeError): + obj = {} + if not isinstance(obj, dict): + obj = {} + prompt = obj.get("prompt") or "" + cwd = obj.get("cwd") + if not should_recall(prompt, env): + return "" + notes = _recall( + prompt, + vault=vault, + index=index, + embedder_name=name, + k=resolve_auto_recall_k(env), + scope=resolve_auto_recall_scope(env), + cwd=cwd, + ) + block = format_block(notes) + return json.dumps(build_hook_output(block)) if block else "" + except Exception: + return "" diff --git a/tests/test_config_auto_recall.py b/tests/test_config_auto_recall.py new file mode 100644 index 0000000..d6a72bc --- /dev/null +++ b/tests/test_config_auto_recall.py @@ -0,0 +1,44 @@ +# SPDX-License-Identifier: Apache-2.0 +from cairn.config import ( + resolve_auto_recall, + resolve_auto_recall_k, + resolve_auto_recall_scope, +) + + +def test_auto_recall_default_on(): + assert resolve_auto_recall(env={}) is True + + +def test_auto_recall_off(): + assert resolve_auto_recall(env={"CAIRN_AUTO_RECALL": "0"}) is False + assert resolve_auto_recall(env={"CAIRN_AUTO_RECALL": "false"}) is False + + +def test_auto_recall_explicit_true(): + assert resolve_auto_recall(env={"CAIRN_AUTO_RECALL": "true"}) is True + assert resolve_auto_recall(env={"CAIRN_AUTO_RECALL": "1"}) is True + + +def test_auto_recall_bad_value_falls_back_true(): + assert resolve_auto_recall(env={"CAIRN_AUTO_RECALL": "maybe"}) is True + + +def test_auto_recall_k_default(): + assert resolve_auto_recall_k(env={}) == 3 + + +def test_auto_recall_k_override(): + assert resolve_auto_recall_k(env={"CAIRN_AUTO_RECALL_K": "5"}) == 5 + + +def test_auto_recall_k_bad_falls_back(): + assert resolve_auto_recall_k(env={"CAIRN_AUTO_RECALL_K": "lots"}) == 3 + + +def test_auto_recall_scope_default(): + assert resolve_auto_recall_scope(env={}) == "all" + + +def test_auto_recall_scope_override_lowercased(): + assert resolve_auto_recall_scope(env={"CAIRN_AUTO_RECALL_SCOPE": "Project"}) == "project" diff --git a/tests/test_recall_hook.py b/tests/test_recall_hook.py new file mode 100644 index 0000000..dba133b --- /dev/null +++ b/tests/test_recall_hook.py @@ -0,0 +1,133 @@ +# SPDX-License-Identifier: Apache-2.0 +import json +from pathlib import Path + +from cairn.embed import get_embedder +from cairn.recall_hook import build_hook_output, format_block, run, should_recall +from tests.search.test_engine import build_index + + +def _idx(tmp_path) -> Path: + return Path(build_index(tmp_path, get_embedder("fake"))) + + +def test_should_recall_gate(): + assert should_recall("how do I brew coffee beans?", env={}) is True + assert should_recall("go", env={}) is False + assert should_recall(" yes ", env={}) is False + assert should_recall("how do I brew coffee?", env={"CAIRN_AUTO_RECALL": "0"}) is False + + +def test_format_block_empty_returns_empty(): + assert format_block([]) == "" + assert format_block([{"permalink": "x", "text": " "}]) == "" + + +def test_format_block_includes_permalink(): + block = format_block([{"permalink": "coffee", "text": "Arabica beans."}]) + assert block.startswith("## Relevant memories (agentcairn)") + assert "Arabica beans." in block + assert "[[coffee]]" in block + + +def test_build_hook_output_shape(): + assert build_hook_output("hi") == { + "hookSpecificOutput": { + "hookEventName": "UserPromptSubmit", + "additionalContext": "hi", + } + } + + +def test_run_injects_relevant_memory(tmp_path): + out = run( + json.dumps({"prompt": "how do I brew coffee beans?"}), + index=_idx(tmp_path), + embedder_name="fake", + env={}, + ) + assert out + data = json.loads(out) + assert data["hookSpecificOutput"]["hookEventName"] == "UserPromptSubmit" + assert "coffee" in data["hookSpecificOutput"]["additionalContext"].lower() + + +def test_run_skips_trivial_prompt(tmp_path): + out = run(json.dumps({"prompt": "go"}), index=_idx(tmp_path), embedder_name="fake", env={}) + assert out == "" + + +def test_run_disabled_via_env(tmp_path): + out = run( + json.dumps({"prompt": "how do I brew coffee beans?"}), + index=_idx(tmp_path), + embedder_name="fake", + env={"CAIRN_AUTO_RECALL": "0"}, + ) + assert out == "" + + +def test_run_no_index_is_silent(tmp_path): + out = run( + json.dumps({"prompt": "how do I brew coffee beans?"}), + index=tmp_path / "missing.duckdb", + embedder_name="fake", + env={}, + ) + assert out == "" + + +def test_run_malformed_stdin_is_silent(tmp_path): + out = run("not json at all", index=_idx(tmp_path), embedder_name="fake", env={}) + assert out == "" + + +def test_run_honors_cairn_embedder_env(tmp_path): + """No explicit embedder_name → CAIRN_EMBEDDER env var is used.""" + out = run( + json.dumps({"prompt": "how do I brew coffee beans?"}), + index=_idx(tmp_path), + env={"CAIRN_EMBEDDER": "fake"}, + ) + assert out + data = json.loads(out) + assert "coffee" in data["hookSpecificOutput"]["additionalContext"].lower() + + +def test_run_valid_non_dict_json_is_silent(tmp_path): + """Valid JSON that is not a dict (string, list) should return ''.""" + idx = _idx(tmp_path) + assert run('"hi"', index=idx, embedder_name="fake", env={}) == "" + assert run("[1, 2]", index=idx, embedder_name="fake", env={}) == "" + + +def test_run_bm25_fallback_when_embedder_none(tmp_path): + """embedder_name='none' → BM25-only path; still returns hits for a keyword query.""" + out = run( + json.dumps({"prompt": "coffee beans"}), + index=_idx(tmp_path), + embedder_name="none", + env={}, + ) + assert out != "" + + +def test_run_uses_payload_cwd_for_project(tmp_path, monkeypatch): + """Project boost/scope resolves from the hook payload's cwd, not the process cwd.""" + from cairn import recall_hook as rh + from cairn.ingest.events import project_from_cwd + + captured: dict = {} + + def fake_search(con, query, **kw): + captured.update(kw) + return [] + + monkeypatch.setattr(rh, "search", fake_search) + rh.run( + json.dumps({"prompt": "how do I brew coffee beans?", "cwd": "/work/acme-api"}), + index=_idx(tmp_path), + embedder_name="fake", + env={}, + ) + assert captured.get("project") == project_from_cwd("/work/acme-api") diff --git a/tests/test_recall_hook_cli.py b/tests/test_recall_hook_cli.py new file mode 100644 index 0000000..a183f0b --- /dev/null +++ b/tests/test_recall_hook_cli.py @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: Apache-2.0 +import json + +from typer.testing import CliRunner + +from cairn.cli import app +from cairn.embed import get_embedder +from tests.search.test_engine import build_index + +runner = CliRunner() + + +def test_recall_hook_cli_stdin_wiring(tmp_path): + idx = build_index(tmp_path, get_embedder("fake")) + r = runner.invoke( + app, + ["recall-hook", "--index", str(idx), "--embedder", "fake"], + input=json.dumps({"prompt": "how do I brew coffee beans?"}), + ) + assert r.exit_code == 0 + data = json.loads(r.output) + assert data["hookSpecificOutput"]["hookEventName"] == "UserPromptSubmit" + + +def test_recall_hook_cli_trivial_prompt_no_output(tmp_path): + idx = build_index(tmp_path, get_embedder("fake")) + r = runner.invoke( + app, + ["recall-hook", "--index", str(idx), "--embedder", "fake"], + input=json.dumps({"prompt": "go"}), + ) + assert r.exit_code == 0 + assert r.output.strip() == ""