diff --git a/CHANGELOG.md b/CHANGELOG.md index a285846c..5cf850d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.0.3](https://github.com/ScriptedAlchemy/tracedecay/compare/v0.0.2...v0.0.3) - 2026-06-19 + +### Fixed + +- *(deps)* remove unused dependencies + +### Other + +- stabilize Windows GC test and cancel stale runs + ## [0.0.2] - 2026-06-18 diff --git a/Cargo.lock b/Cargo.lock index 7ad82028..c79fbc47 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4499,7 +4499,7 @@ checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" [[package]] name = "tracedecay" -version = "0.0.2" +version = "0.0.3" dependencies = [ "amari-holographic", "axum 0.8.9", diff --git a/Cargo.toml b/Cargo.toml index e2459d6f..c61b3839 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "tracedecay" -version = "0.0.2" +version = "0.0.3" edition = "2021" description = "Code intelligence tool that builds a semantic knowledge graph from Rust, Go, Java, Scala, TypeScript, Python, C, C++, Kotlin, C#, Swift, and many more codebases" license = "MIT" diff --git a/docs/superpowers/plans/2026-03-27-daemon-mode.md b/docs/superpowers/plans/2026-03-27-daemon-mode.md deleted file mode 100644 index 5e3d57e8..00000000 --- a/docs/superpowers/plans/2026-03-27-daemon-mode.md +++ /dev/null @@ -1,726 +0,0 @@ -# Daemon Mode Implementation Plan - -> **Rebrand note:** The project has since been renamed **TraceDecay** (binary/crate `tracedecay`, MCP tools `tracedecay_*`). This dated planning artifact keeps the TokenSave-era names it was written with. - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** A background daemon that watches all tracked tokensave projects for file changes and automatically runs incremental syncs. - -**Architecture:** A new `tokensave daemon` subcommand backed by `src/daemon.rs`. Uses `notify` for filesystem watching, tokio timers for per-project debounce, and `daemon-kit` for cross-platform daemonization (fork on Unix via daemonize2, Windows Service via windows-service), PID file management, and service installation (launchd/systemd/Windows Service). Discovers projects from the global DB, re-polls every 60s for new ones. - -**Tech Stack:** Rust, `daemon-kit` (cross-platform daemon/service), `notify` v7 (file watcher), tokio (async runtime, timers), existing `TokenSave::sync()`, existing `GlobalDb`. - ---- - -## File Map - -| Action | File | Responsibility | -|--------|------|----------------| -| Modify | `Cargo.toml` | Add `daemon-kit` and `notify` dependencies | -| Modify | `src/user_config.rs` | Add `daemon_debounce: String` field | -| Modify | `src/global_db.rs` | Add `list_project_paths()` method | -| Create | `src/daemon.rs` | Core daemon: watcher, debounce, sync loop (uses daemon-kit for daemonize/PID/service) | -| Modify | `src/lib.rs` | Add `pub mod daemon;` | -| Modify | `src/main.rs` | Add `Commands::Daemon` variant and handler | -| Modify | `src/doctor.rs` | Add daemon running/autostart checks | -| Modify | `tests/user_config_test.rs` | Add `daemon_debounce` to round-trip test | - ---- - -### Task 1: Add dependencies and `daemon_debounce` config field - -**Files:** -- Modify: `Cargo.toml` -- Modify: `src/user_config.rs` -- Modify: `tests/user_config_test.rs` - -- [ ] **Step 1: Add daemon-kit and notify to Cargo.toml** - -In `[dependencies]`, add: -```toml -daemon-kit = "0.1" -notify = { version = "7", default-features = false, features = ["macos_fsevent"] } -``` - -- [ ] **Step 2: Add `daemon_debounce` to UserConfig** - -In `src/user_config.rs`, add after the `installed_agents` field: -```rust -/// Debounce duration for the daemon file watcher (e.g. "15s", "1m"). -#[serde(default = "default_daemon_debounce")] -pub daemon_debounce: String, -``` - -Add the default function: -```rust -fn default_daemon_debounce() -> String { - "15s".to_string() -} -``` - -In the `Default` impl, add: -```rust -daemon_debounce: default_daemon_debounce(), -``` - -- [ ] **Step 3: Update round-trip test** - -In `tests/user_config_test.rs`, add `daemon_debounce: "30s".to_string()` to the test struct. - -- [ ] **Step 4: Build and test** - -Run: `cargo build && cargo test user_config` - -- [ ] **Step 5: Commit** -``` -feat: add notify/nix deps and daemon_debounce config field -``` - ---- - -### Task 2: Add `list_project_paths()` to GlobalDb - -**Files:** -- Modify: `src/global_db.rs` - -- [ ] **Step 1: Add the method** - -Add to `impl GlobalDb`: -```rust -/// Returns all tracked project paths. -pub async fn list_project_paths(&self) -> Vec { - let mut rows = match self - .conn - .query("SELECT path FROM projects", ()) - .await - { - Ok(r) => r, - Err(_) => return Vec::new(), - }; - let mut paths = Vec::new(); - loop { - match rows.next().await { - Ok(Some(row)) => { - if let Ok(path) = row.get::(0) { - paths.push(path); - } - } - _ => break, - } - } - paths -} -``` - -- [ ] **Step 2: Build** - -Run: `cargo build` - -- [ ] **Step 3: Commit** -``` -feat: add GlobalDb::list_project_paths() -``` - ---- - -### Task 3: Create `src/daemon.rs` — duration parser and daemon-kit setup - -**Files:** -- Create: `src/daemon.rs` -- Modify: `src/lib.rs` - -- [ ] **Step 1: Create daemon.rs with duration parser and daemon-kit Daemon instance** - -```rust -//! Background daemon that watches all tracked tokensave projects for file -//! changes and runs incremental syncs automatically. - -use std::path::PathBuf; -use std::time::Duration; - -use daemon_kit::{Daemon, DaemonConfig}; - -use crate::errors::{Result, TokenSaveError}; - -/// Parse a human-readable duration string like "15s" or "1m" into a Duration. -/// Returns None if the format is unrecognized. -pub fn parse_duration(s: &str) -> Option { - let s = s.trim(); - if let Some(secs) = s.strip_suffix('s') { - secs.trim().parse::().ok().map(Duration::from_secs) - } else if let Some(mins) = s.strip_suffix('m') { - mins.trim().parse::().ok().map(|m| Duration::from_secs(m * 60)) - } else { - s.parse::().ok().map(Duration::from_secs) - } -} - -/// Build the daemon-kit Daemon instance with tokensave paths. -fn build_daemon() -> std::result::Result { - let home = dirs::home_dir().ok_or_else(|| TokenSaveError::Config { - message: "cannot determine home directory".to_string(), - })?; - let ts_dir = home.join(".tokensave"); - let bin = crate::agents::which_tokensave().unwrap_or_else(|| "tokensave".to_string()); - - let config = DaemonConfig::new("tokensave-daemon") - .pid_dir(&ts_dir) - .log_file(ts_dir.join("daemon.log")) - .executable(PathBuf::from(bin)) - .service_args(vec!["daemon".to_string(), "--foreground".to_string()]) - .description("tokensave file watcher daemon"); - - Ok(Daemon::new(config)) -} - -/// Returns the PID of the running daemon, or None. -pub fn running_daemon_pid() -> Option { - build_daemon().ok()?.running_pid() -} - -/// Returns true if an autostart service is installed. -pub fn is_autostart_enabled() -> bool { - build_daemon().ok().is_some_and(|d| d.is_service_installed()) -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn parse_duration_seconds() { - assert_eq!(parse_duration("15s"), Some(Duration::from_secs(15))); - assert_eq!(parse_duration("30s"), Some(Duration::from_secs(30))); - assert_eq!(parse_duration(" 5s "), Some(Duration::from_secs(5))); - } - - #[test] - fn parse_duration_minutes() { - assert_eq!(parse_duration("1m"), Some(Duration::from_secs(60))); - assert_eq!(parse_duration("2m"), Some(Duration::from_secs(120))); - } - - #[test] - fn parse_duration_bare_number() { - assert_eq!(parse_duration("10"), Some(Duration::from_secs(10))); - } - - #[test] - fn parse_duration_invalid() { - assert_eq!(parse_duration("abc"), None); - assert_eq!(parse_duration(""), None); - assert_eq!(parse_duration("1h"), None); - } -} -``` - -- [ ] **Step 2: Add `pub mod daemon;` to lib.rs** - -In `src/lib.rs`, add: -```rust -pub mod daemon; -``` - -- [ ] **Step 3: Build and test** - -Run: `cargo build && cargo test daemon` - -- [ ] **Step 4: Commit** -``` -feat: daemon duration parser and daemon-kit setup -``` - ---- - -### Task 4: Daemon core event loop - -**Files:** -- Modify: `src/daemon.rs` - -- [ ] **Step 1: Add the main daemon run function** - -Append to `src/daemon.rs` — this is the core loop. It: -1. Opens the global DB -2. Reads all project paths -3. Sets up `notify` watchers for each -4. Runs a tokio select loop: file events → mark dirty + reset debounce timer; 60s ticker → re-poll global DB for new projects; debounce timer fires → sync the dirty project; SIGTERM/SIGINT → shutdown. - -```rust -use std::collections::{HashMap, HashSet}; -use std::sync::Arc; -use tokio::sync::mpsc; -use tokio::time::{self, Instant}; -use notify::{RecommendedWatcher, RecursiveMode, Watcher, Event}; - -/// Directories to ignore inside watched projects. -const IGNORED_DIRS: &[&str] = &[ - ".tokensave", ".git", "node_modules", "target", ".build", - "__pycache__", ".next", "dist", "build", ".cache", -]; - -/// Run the daemon event loop. This function does not return until -/// a shutdown signal is received. -pub async fn run(foreground: bool) -> Result<()> { - if let Some(pid) = running_daemon_pid() { - return Err(TokenSaveError::Config { - message: format!("daemon already running (PID: {pid})"), - }); - } - - if !foreground { - daemonize()?; - } - - write_pid_file()?; - - // Set up graceful shutdown on SIGTERM/SIGINT - let shutdown = tokio::signal::ctrl_c(); - - let config = crate::user_config::UserConfig::load(); - let debounce = parse_duration(&config.daemon_debounce) - .unwrap_or(Duration::from_secs(15)); - - let result = run_loop(debounce, shutdown).await; - - remove_pid_file(); - result -} - -async fn run_loop( - debounce: Duration, - shutdown: impl std::future::Future>, -) -> Result<()> { - let (tx, mut rx) = mpsc::channel::(256); - - let mut watchers: HashMap = HashMap::new(); - let mut dirty: HashMap = HashMap::new(); - - // Initial project discovery - let project_paths = discover_projects().await; - for path in &project_paths { - if let Some(w) = create_watcher(path, tx.clone()) { - watchers.insert(path.clone(), w); - } - } - - daemon_log(&format!("started, watching {} projects", watchers.len())); - - let mut discovery_interval = time::interval(Duration::from_secs(60)); - discovery_interval.tick().await; // consume first immediate tick - - tokio::pin!(shutdown); - - loop { - // Find the next debounce deadline - let next_deadline = dirty.values().copied().min(); - let sleep = match next_deadline { - Some(deadline) => tokio::time::sleep_until(deadline), - None => tokio::time::sleep(Duration::from_secs(3600)), // park - }; - tokio::pin!(sleep); - - tokio::select! { - _ = &mut shutdown => { - daemon_log("shutting down (signal)"); - break; - } - Some(project_root) = rx.recv() => { - // File change in a project — mark dirty, reset timer - dirty.insert(project_root, Instant::now() + debounce); - } - _ = &mut sleep, if next_deadline.is_some() => { - // A debounce timer fired — sync all projects past their deadline - let now = Instant::now(); - let ready: Vec = dirty - .iter() - .filter(|(_, deadline)| **deadline <= now) - .map(|(path, _)| path.clone()) - .collect(); - for path in ready { - dirty.remove(&path); - sync_project(&path).await; - } - } - _ = discovery_interval.tick() => { - // Re-discover projects - let current = discover_projects().await; - let current_set: HashSet<&PathBuf> = current.iter().collect(); - let watched_set: HashSet<&PathBuf> = watchers.keys().collect(); - - // Add new projects - for path in current_set.difference(&watched_set) { - if let Some(w) = create_watcher(path, tx.clone()) { - daemon_log(&format!("discovered new project: {}", path.display())); - watchers.insert((*path).clone(), w); - } - } - // Remove stale projects - let stale: Vec = watched_set - .difference(¤t_set) - .map(|p| (*p).clone()) - .collect(); - for path in stale { - watchers.remove(&path); - dirty.remove(&path); - } - } - } - } - - Ok(()) -} - -/// Query the global DB for all tracked project paths. -async fn discover_projects() -> Vec { - let Some(gdb) = crate::global_db::GlobalDb::open().await else { - return Vec::new(); - }; - gdb.list_project_paths() - .await - .into_iter() - .filter_map(|s| { - let p = PathBuf::from(&s); - if p.is_dir() && crate::tokensave::TokenSave::is_initialized(&p) { - Some(p) - } else { - None - } - }) - .collect() -} - -/// Create a notify watcher for a project root. Sends the project root -/// path to `tx` on any relevant file event. -fn create_watcher(project_root: &Path, tx: mpsc::Sender) -> Option { - let root = project_root.to_path_buf(); - let mut watcher = notify::recommended_watcher(move |res: std::result::Result| { - let Ok(event) = res else { return }; - // Only care about create/modify/remove - if !matches!( - event.kind, - notify::EventKind::Create(_) - | notify::EventKind::Modify(_) - | notify::EventKind::Remove(_) - ) { - return; - } - // Check if any path in the event should be ignored - let dominated_by_ignored = event.paths.iter().all(|p| { - p.components().any(|c| { - IGNORED_DIRS.contains(&c.as_os_str().to_str().unwrap_or("")) - }) - }); - if dominated_by_ignored { - return; - } - let _ = tx.blocking_send(root.clone()); - }) - .ok()?; - watcher.watch(project_root, RecursiveMode::Recursive).ok()?; - Some(watcher) -} - -/// Run an incremental sync on a single project. Best-effort. -async fn sync_project(project_root: &Path) { - let start = std::time::Instant::now(); - let Ok(cg) = crate::tokensave::TokenSave::open(project_root).await else { - daemon_log(&format!("failed to open {}", project_root.display())); - return; - }; - match cg.sync().await { - Ok(result) => { - let ms = start.elapsed().as_millis(); - daemon_log(&format!( - "synced {} — {} added, {} modified, {} removed ({}ms)", - project_root.display(), - result.files_added, - result.files_modified, - result.files_removed, - ms - )); - // Best-effort update global DB - if let Some(gdb) = crate::global_db::GlobalDb::open().await { - let tokens = cg.get_tokens_saved().await.unwrap_or(0); - gdb.upsert(project_root, tokens).await; - } - } - Err(e) => { - daemon_log(&format!("sync failed for {}: {e}", project_root.display())); - } - } -} - -/// Append a timestamped line to the daemon log file. -fn daemon_log(msg: &str) { - let now = chrono::Local::now().format("%Y-%m-%d %H:%M:%S"); - let line = format!("[{now}] {msg}\n"); - // Also print to stderr if running in foreground - eprint!("{line}"); - if let Some(log_path) = log_file_path() { - use std::io::Write; - if let Ok(mut f) = std::fs::OpenOptions::new() - .create(true) - .append(true) - .open(log_path) - { - let _ = f.write_all(line.as_bytes()); - } - } -} -``` - -Wait — this uses `chrono`. Let me avoid adding a new dep and just use `std::time` for the log timestamp. Replace the `daemon_log` function: - -```rust -/// Append a timestamped line to the daemon log file. -fn daemon_log(msg: &str) { - let secs = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default() - .as_secs(); - let line = format!("[{secs}] {msg}\n"); - eprint!("{line}"); - if let Some(log_path) = log_file_path() { - use std::io::Write; - if let Ok(mut f) = std::fs::OpenOptions::new() - .create(true) - .append(true) - .open(log_path) - { - let _ = f.write_all(line.as_bytes()); - } - } -} -``` - -- [ ] **Step 2: Build** - -Run: `cargo build` - -Note: `notify` v7 will be pulled in from Cargo.toml. If there are API differences (v7 uses `Event` directly, not `DebouncedEvent`), adjust the `create_watcher` call accordingly. - -- [ ] **Step 3: Commit** -``` -feat: daemon core event loop with file watching and debounced sync -``` - ---- - -### Task 5: Daemon run/stop/status/autostart via daemon-kit - -**Files:** -- Modify: `src/daemon.rs` - -All daemonization, PID management, stop/status, and service installation is delegated to the `daemon-kit` crate. Add these thin wrapper functions: - -- [ ] **Step 1: Add run, stop, status, enable/disable_autostart functions** - -```rust -/// Start the daemon. Forks to background on Unix unless `foreground` is true. -/// On Windows, runs as a Windows Service. -pub async fn run(foreground: bool) -> Result<()> { - let daemon = build_daemon()?; - - let config = crate::user_config::UserConfig::load(); - let debounce = parse_duration(&config.daemon_debounce) - .unwrap_or(Duration::from_secs(15)); - - // daemon-kit handles fork/PID/detach; we pass it the event loop closure - daemon - .start(foreground, move || { - // Build a new tokio runtime inside the forked child - let rt = tokio::runtime::Runtime::new().map_err(|e| { - daemon_kit::DaemonError::Daemonize(format!("failed to create runtime: {e}")) - })?; - rt.block_on(async { - run_loop(debounce).await.map_err(|e| { - daemon_kit::DaemonError::Daemonize(e.to_string()) - }) - }) - }) - .map_err(|e| TokenSaveError::Config { - message: format!("daemon error: {e}"), - }) -} - -/// Stop the running daemon. -pub fn stop() -> Result<()> { - let daemon = build_daemon()?; - daemon.stop().map_err(|e| TokenSaveError::Config { - message: format!("{e}"), - })?; - eprintln!("tokensave daemon stopped"); - Ok(()) -} - -/// Print daemon status and return exit code (0 = running, 1 = not running). -pub fn status() -> i32 { - match running_daemon_pid() { - Some(pid) => { - eprintln!("tokensave daemon is running (PID: {pid})"); - 0 - } - None => { - eprintln!("tokensave daemon is not running"); - 1 - } - } -} - -/// Install autostart service (launchd/systemd/Windows Service). -pub fn enable_autostart() -> Result<()> { - let daemon = build_daemon()?; - daemon.install_service().map_err(|e| TokenSaveError::Config { - message: format!("{e}"), - })?; - eprintln!("\x1b[32m✔\x1b[0m Autostart service installed"); - Ok(()) -} - -/// Remove autostart service. -pub fn disable_autostart() -> Result<()> { - let daemon = build_daemon()?; - daemon.uninstall_service().map_err(|e| TokenSaveError::Config { - message: format!("{e}"), - })?; - eprintln!("\x1b[32m✔\x1b[0m Autostart service removed"); - Ok(()) -} -``` - -- [ ] **Step 2: Build** - -Run: `cargo build` - -- [ ] **Step 3: Commit** -``` -feat: daemon run/stop/status/autostart via daemon-kit -``` - ---- - -### Task 6: CLI integration and doctor checks - -**Files:** -- Modify: `src/main.rs` -- Modify: `src/doctor.rs` - -- [ ] **Step 1: Add Commands::Daemon to main.rs** - -In the `Commands` enum, add: -```rust -/// Background file watcher daemon -Daemon { - /// Run in foreground (don't fork) - #[arg(long)] - foreground: bool, - /// Stop the running daemon - #[arg(long)] - stop: bool, - /// Show daemon status - #[arg(long)] - status: bool, - /// Install autostart service (launchd/systemd) - #[arg(long)] - enable_autostart: bool, - /// Remove autostart service - #[arg(long)] - disable_autostart: bool, -}, -``` - -- [ ] **Step 2: Add handler in the match block** - -In the `match command { ... }` block, add: -```rust -Commands::Daemon { foreground, stop, status, enable_autostart, disable_autostart } => { - if stop { - tokensave::daemon::stop()?; - } else if status { - let code = tokensave::daemon::status(); - std::process::exit(code); - } else if enable_autostart { - tokensave::daemon::enable_autostart()?; - } else if disable_autostart { - tokensave::daemon::disable_autostart()?; - } else { - tokensave::daemon::run(foreground).await?; - } -} -``` - -- [ ] **Step 3: Add daemon checks to doctor.rs** - -Add a new function `check_daemon` and call it from `run_doctor`: - -```rust -fn check_daemon(dc: &mut DoctorCounters) { - eprintln!("\n\x1b[1mDaemon\x1b[0m"); - match crate::daemon::running_daemon_pid() { - Some(pid) => dc.pass(&format!("Daemon is running (PID: {pid})")), - None => dc.warn("Daemon is not running — run `tokensave daemon` to start"), - } - if crate::daemon::is_autostart_enabled() { - #[cfg(target_os = "macos")] - dc.pass("Autostart enabled (launchd)"); - #[cfg(target_os = "linux")] - dc.pass("Autostart enabled (systemd)"); - #[cfg(not(any(target_os = "macos", target_os = "linux")))] - dc.pass("Autostart enabled"); - } else { - dc.warn("Autostart not configured — run `tokensave daemon --enable-autostart`"); - } -} -``` - -In `run_doctor`, add `check_daemon(&mut dc);` before the network checks. - -- [ ] **Step 4: Build and test** - -Run: `cargo build && cargo test` - -- [ ] **Step 5: Manual test** - -```bash -./target/debug/tokensave daemon --status -./target/debug/tokensave daemon --foreground & -./target/debug/tokensave daemon --status -./target/debug/tokensave daemon --stop -./target/debug/tokensave doctor | grep -A2 Daemon -``` - -- [ ] **Step 6: Commit** -``` -feat: daemon CLI subcommand and doctor integration -``` - ---- - -### Task 7: CHANGELOG and final verification - -**Files:** -- Modify: `CHANGELOG.md` - -- [ ] **Step 1: Add CHANGELOG entry** - -Add a new `## [Unreleased]` or version section with: -```markdown -### Added -- **Daemon mode** — `tokensave daemon` watches all tracked projects for file changes and runs incremental syncs automatically; debounce configurable via `daemon_debounce` in `~/.tokensave/config.toml` (default `"15s"`) -- **Daemon management** — `--stop`, `--status`, `--foreground` flags for process control; PID file at `~/.tokensave/daemon.pid` -- **Autostart service** — `--enable-autostart` / `--disable-autostart` generates and manages a launchd plist (macOS) or systemd user unit (Linux) -- **Doctor daemon checks** — `tokensave doctor` now reports daemon running status and autostart configuration -``` - -- [ ] **Step 2: Full test suite** - -Run: `cargo test` - -- [ ] **Step 3: Release build** - -Run: `cargo build --release` - -- [ ] **Step 4: Commit** -``` -feat: daemon mode — background file watcher with auto-sync -``` diff --git a/docs/superpowers/plans/2026-06-09-tokensave-lcm-session-rewrite.md b/docs/superpowers/plans/2026-06-09-tokensave-lcm-session-rewrite.md deleted file mode 100644 index 76a34514..00000000 --- a/docs/superpowers/plans/2026-06-09-tokensave-lcm-session-rewrite.md +++ /dev/null @@ -1,1748 +0,0 @@ -# TokenSave LCM Session Rewrite Implementation Plan - -> **Rebrand note:** The project has since been renamed **TraceDecay** (binary/crate `tracedecay`, MCP tools `tracedecay_*`). This dated planning artifact keeps the TokenSave-era names it was written with; read `tokensave` / `tokensave_*` as `tracedecay` / `tracedecay_*` when applying it to the current codebase. - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Rewrite TokenSave's existing session internals into a lossless, LCM-grade session store inside the existing `sessions.db`, while preserving compatible `tokensave_message_search` behavior and adding deterministic LCM load/expand/status/compression APIs. - -**Architecture:** Rust owns the durable LCM store, idempotent schema migrations, raw-message and summary-DAG state, bounded derived indexes/snippets, storage path containment, and public CLI/MCP JSON contracts. The generated Hermes Python plugin owns Hermes `ContextEngine` lifecycle registration and Hermes auxiliary LLM calls, calling Rust through the existing `tokensave tool ... --json --args` subprocess bridge. Codegraph `ContextBuilder` and `src/memory/*` fact storage stay separate from session compression and are only regression-tested for non-coupling. - -**Tech Stack:** Rust 1.95, `libsql`/SQLite FTS5, `tokio`, `serde`/`serde_json`, generated Hermes Python plugin files, `python3 -m py_compile`, `cargo test`, `cargo fmt --all -- --check`, and `cargo clippy --all-targets`. - ---- - -## Source Anchors - -- Approved design: `docs/superpowers/specs/2026-06-09-tokensave-lcm-session-rewrite-design.md`. -- Current DB/session write path: `src/global_db.rs`, `src/sessions/mod.rs`, `src/sessions/source.rs`, `src/sessions/cursor.rs`. -- Current search tool: `src/mcp/tools/handlers/session.rs`, `src/mcp/tools/definitions.rs`, `src/mcp/tools/handlers/mod.rs`. -- Current Hermes generator and subprocess bridge: `src/agents/hermes.rs`, `src/tool_command.rs`. -- Existing tests to evolve: `tests/session_global_db_test.rs`, `tests/mcp_handler_test.rs`, `tests/agent_test.rs`. -- Current authoritative cap to remove from new writes: `MAX_SESSION_MESSAGE_TEXT_BYTES` in `src/global_db.rs`; after this plan, any cap belongs only to derived snippets, FTS/index text, MCP response truncation, or bounded rendering. - -## File Structure Map - -Create these focused Rust modules: - -- `src/sessions/lcm/mod.rs` - public module boundary, exports types and store helpers. -- `src/sessions/lcm/types.rs` - stable Rust structs/enums for raw messages, payload refs, summary nodes, lifecycle state, query inputs, and JSON outputs. -- `src/sessions/lcm/schema.rs` - schema version constants and idempotent migration DDL for LCM tables inside the existing `sessions.db`. -- `src/sessions/lcm/store.rs` - `LcmStore<'db>` wrapper that binds `GlobalDb` plus an explicit storage root and exposes raw ingest, query, DAG, payload, and compression operations. -- `src/sessions/lcm/raw.rs` - lossless raw-message ingest, compatibility projection writes, legacy-row carry-forward, and authoritative content loading. -- `src/sessions/lcm/payload.rs` - externalized payload creation, hashing, permissions, basename-only refs, root containment, and paginated expansion. -- `src/sessions/lcm/dag.rs` - summary DAG persistence, source lineage, subtree expansion, and depth/window metadata. -- `src/sessions/lcm/query.rs` - search/load/describe/status query assembly, stable cursors, bounded snippets, and JSON result types. -- `src/sessions/lcm/compression.rs` - deterministic compression lifecycle primitives, fake summarizer injection, frontier/debt state transitions, and active-context assembly. -- `src/sessions/lcm/security.rs` - ingest-protection classification for large/binary-ish payloads, data URI/base64 detection, sensitive-redaction metadata, and integrity scan helpers. -- `src/sessions/lcm/hermes.rs` - Rust JSON request/response contracts used by the generated Hermes `ContextEngine` adapter. - -Modify these existing Rust files: - -- `src/sessions/mod.rs` - add `pub mod lcm;` and re-export the public LCM types required by handlers/tests. -- `src/global_db.rs` - call the LCM schema migration during `GlobalDb::open_at`, expose a crate-private connection accessor for LCM modules, and change `upsert_session_message` so new authoritative content is lossless while compatibility/index fields remain bounded. -- `src/sessions/source.rs` - route parsed provider messages through the LCM raw ingest path while preserving incremental parse offsets and provider-normalized session metadata. -- `src/sessions/cursor.rs` - keep `project_session_db_path(project_root).join("sessions.db")` behavior and pass the project-local `.tokensave` storage root into `LcmStore`. -- `src/mcp/tools/definitions.rs` - add additive LCM tool schemas for `tokensave_lcm_status`, `tokensave_lcm_load_session`, `tokensave_lcm_grep`, `tokensave_lcm_describe`, `tokensave_lcm_expand`, `tokensave_lcm_expand_query`, `tokensave_lcm_preflight`, and `tokensave_lcm_compress`. -- `src/mcp/tools/handlers/session.rs` - keep `handle_message_search` compatible and add LCM handlers that call deterministic Rust APIs. -- `src/mcp/tools/handlers/mod.rs` - register the new LCM handler names in `handle_tool_call`. -- `src/agents/hermes.rs` - generate the Hermes `ContextEngine` adapter, LCM bridge helpers, local/profile storage configuration, auxiliary summarizer calls, reasoning stripping, and Python tests embedded in Rust fixtures. -- `src/main.rs` - only if install flags need to pass explicit Hermes profile/locality metadata into `HermesIntegration`; otherwise leave unchanged. -- `src/tool_command.rs` - only if LCM bridge tests need stronger `--project` coverage for subprocess calls; keep the existing `--json --args` contract. - -Create these tests: - -- `tests/session_lcm_schema_test.rs` - schema migration, idempotency, legacy carry-forward, and single-DB assertions. -- `tests/session_lcm_raw_test.rs` - lossless raw ingest, capped derived snippets, compatibility projections, and authoritative load. -- `tests/session_lcm_payload_test.rs` - payload externalization, path containment, permissions, hashes, missing/unreferenced scans, and cross-session denial. -- `tests/session_lcm_dag_test.rs` - summary DAG persistence, source lineage, subtree expansion, and restart recovery. -- `tests/session_lcm_query_test.rs` - load/grep/describe/status APIs and deterministic pagination. -- `tests/session_lcm_compression_test.rs` - lifecycle/frontier/debt primitives with deterministic fake summarizers. -- `tests/session_lcm_ingest_protection_test.rs` - data URI/base64/large tool output protection and bounded index text. -- `tests/hermes_lcm_bridge_test.rs` - generated Python context engine, subprocess bridge calls, no-op/fake summarizer behavior, auxiliary LLM routing, reasoning stripping, and fallback behavior. - -Modify these tests: - -- `tests/session_global_db_test.rs` - replace capped-authoritative assertions with lossless storage assertions and keep search/filter compatibility. -- `tests/mcp_handler_test.rs` - assert new LCM tool definitions, handler dispatch, and unchanged `tokensave_message_search` provider/scope schema. -- `tests/agent_test.rs` - extend Hermes generated-plugin tests for `ContextEngine` registration, local/profile storage locality, Python compile checks, and non-overwrite memory provider behavior. - -Do not modify these production areas except for regression tests proving separation: - -- `src/context/builder.rs` - codegraph context retrieval remains independent from LCM. -- `src/memory/*` - fact memory remains independent from summary DAG state. - -## Cross-Cutting Invariants - -- The existing `sessions.db` is the only primary TokenSave-managed session database. Migrations evolve it in place; tests must fail if a second primary LCM DB path is introduced. -- New authoritative session content is lossless. `session_messages.text` may become a bounded compatibility projection, but the raw message store must preserve full content inline or through an externalized payload ref. -- Existing rows that were already capped remain best-effort legacy data and must be marked `legacy_truncated = true` when carried into the LCM raw store. -- FTS/index text, snippets, MCP response text, and display previews are derived and bounded. They must never be the only copy of new raw content. -- Storage roots are explicit: project-local installs use `crate::config::get_tokensave_dir(project_root)`; non-local Hermes profile installs use the selected Hermes profile directory plus `.tokensave`. -- Python bridge calls use `tokensave tool --json --args ` and optional `--project ` when the context engine knows the project root. Do not introduce PyO3 or native bindings in this implementation. -- LCM summaries are not memory facts. `summary_nodes` and lifecycle/frontier rows live in the session DB, not in `src/memory/*`. - ---- - -## Task 1: Session Schema Migration Foundation - -**Files:** -- Create: `src/sessions/lcm/mod.rs` -- Create: `src/sessions/lcm/types.rs` -- Create: `src/sessions/lcm/schema.rs` -- Modify: `src/sessions/mod.rs` -- Modify: `src/global_db.rs` -- Create: `tests/session_lcm_schema_test.rs` - -- [ ] **Step 1: Write failing schema migration tests** - -Add tests that open an old-style `sessions.db`, run `GlobalDb::open_at`, and assert the LCM tables exist in that same file. - -```rust -#[tokio::test] -async fn lcm_schema_migrates_legacy_sessions_db_in_place() { - let tmp = tempfile::TempDir::new().unwrap(); - let db_path = tmp.path().join(".tokensave").join("sessions.db"); - std::fs::create_dir_all(db_path.parent().unwrap()).unwrap(); - - let old_db = libsql::Builder::new_local(&db_path).build().await.unwrap(); - let conn = old_db.connect().unwrap(); - conn.execute_batch( - "CREATE TABLE sessions ( - provider TEXT NOT NULL, - session_id TEXT NOT NULL, - project_key TEXT NOT NULL, - project_path TEXT NOT NULL, - title TEXT, - started_at INTEGER, - ended_at INTEGER, - transcript_path TEXT, - metadata_json TEXT, - PRIMARY KEY(provider, session_id) - ); - CREATE TABLE session_messages ( - provider TEXT NOT NULL, - message_id TEXT NOT NULL, - session_id TEXT NOT NULL, - role TEXT NOT NULL, - timestamp INTEGER, - ordinal INTEGER NOT NULL, - text TEXT NOT NULL, - kind TEXT, - model TEXT, - tool_names TEXT, - source_path TEXT, - source_offset INTEGER, - metadata_json TEXT, - PRIMARY KEY(provider, message_id) - ); - INSERT INTO sessions(provider, session_id, project_key, project_path) - VALUES ('cursor', 'legacy-session', '/tmp/project', '/tmp/project'); - INSERT INTO session_messages(provider, message_id, session_id, role, ordinal, text) - VALUES ('cursor', 'legacy-message', 'legacy-session', 'assistant', 1, 'legacy text');", - ).await.unwrap(); - drop(conn); - drop(old_db); - - let db = tokensave::global_db::GlobalDb::open_at(&db_path).await.unwrap(); - assert_eq!(db.lcm_schema_version().await.unwrap(), tokensave::sessions::lcm::LCM_SCHEMA_VERSION); - - let legacy = db - .lcm_load_raw_message("cursor", "legacy-message") - .await - .expect("legacy message should be carried into raw store"); - assert_eq!(legacy.session_id, "legacy-session"); - assert_eq!(legacy.content, "legacy text"); - assert!(legacy.legacy_source); - assert!(!legacy.legacy_truncated); -} -``` - -- [ ] **Step 2: Run the red test** - -Run: `cargo test --test session_lcm_schema_test lcm_schema_migrates_legacy_sessions_db_in_place -- --nocapture` - -Expected: FAIL with missing `tokensave::sessions::lcm`, missing `GlobalDb::lcm_schema_version`, or missing `GlobalDb::lcm_load_raw_message`. - -- [ ] **Step 3: Add LCM module boundary and schema version types** - -Sketch: - -```rust -// src/sessions/lcm/mod.rs -pub mod schema; -pub mod types; - -pub use schema::LCM_SCHEMA_VERSION; -pub use types::{LcmRawMessage, LcmStorageKind}; -``` - -```rust -// src/sessions/lcm/types.rs -#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] -pub struct LcmRawMessage { - pub provider: String, - pub message_id: String, - pub session_id: String, - pub store_id: i64, - pub role: String, - pub ordinal: i64, - pub timestamp: Option, - pub content: String, - pub content_hash: String, - pub storage_kind: LcmStorageKind, - pub payload_ref: Option, - pub legacy_source: bool, - pub legacy_truncated: bool, - pub metadata_json: Option, -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum LcmStorageKind { - Inline, - External, -} -``` - -- [ ] **Step 4: Add idempotent schema migration inside `GlobalDb::open_at`** - -Sketch: - -```rust -// src/sessions/lcm/schema.rs -pub const LCM_SCHEMA_VERSION: i64 = 1; - -pub(crate) async fn ensure_lcm_schema(conn: &libsql::Connection) -> Option<()> { - conn.execute_batch( - "CREATE TABLE IF NOT EXISTS session_schema_migrations ( - name TEXT PRIMARY KEY, - version INTEGER NOT NULL, - applied_at INTEGER NOT NULL DEFAULT (unixepoch()) - ); - CREATE TABLE IF NOT EXISTS lcm_raw_messages ( - provider TEXT NOT NULL, - message_id TEXT NOT NULL, - session_id TEXT NOT NULL, - store_id INTEGER PRIMARY KEY AUTOINCREMENT, - role TEXT NOT NULL, - ordinal INTEGER NOT NULL, - timestamp INTEGER, - content TEXT, - content_hash TEXT NOT NULL, - storage_kind TEXT NOT NULL CHECK(storage_kind IN ('inline', 'external')), - payload_ref TEXT, - snippet_text TEXT NOT NULL, - index_text TEXT NOT NULL, - legacy_source INTEGER NOT NULL DEFAULT 0, - legacy_truncated INTEGER NOT NULL DEFAULT 0, - metadata_json TEXT, - UNIQUE(provider, message_id), - FOREIGN KEY(provider, session_id) REFERENCES sessions(provider, session_id) ON DELETE CASCADE - ); - CREATE INDEX IF NOT EXISTS idx_lcm_raw_session_order - ON lcm_raw_messages(provider, session_id, store_id); - CREATE VIRTUAL TABLE IF NOT EXISTS lcm_raw_messages_fts USING fts5( - index_text, role, metadata_json, - content='lcm_raw_messages', - content_rowid='store_id' - );", - ).await.ok()?; - - carry_forward_legacy_messages(conn).await?; - conn.execute( - "INSERT INTO session_schema_migrations(name, version) - VALUES ('lcm', ?1) - ON CONFLICT(name) DO UPDATE SET version = excluded.version, applied_at = unixepoch()", - libsql::params![LCM_SCHEMA_VERSION], - ).await.ok()?; - Some(()) -} -``` - -Call `crate::sessions::lcm::schema::ensure_lcm_schema(&conn).await?;` immediately after the existing `ensure_session_parent_columns(&conn).await?;` in `GlobalDb::open_at`. - -- [ ] **Step 5: Add introspection helpers used by tests** - -Sketch: - -```rust -impl GlobalDb { - pub async fn lcm_schema_version(&self) -> Option { - let mut rows = self.conn.query( - "SELECT version FROM session_schema_migrations WHERE name = 'lcm'", - (), - ).await.ok()?; - rows.next().await.ok()??.get::(0).ok() - } - - pub async fn lcm_load_raw_message( - &self, - provider: &str, - message_id: &str, - ) -> Option { - crate::sessions::lcm::schema::load_raw_message(&self.conn, provider, message_id).await - } -} -``` - -- [ ] **Step 6: Prove migration idempotency** - -Add `lcm_schema_migration_is_idempotent` in `tests/session_lcm_schema_test.rs` that calls `GlobalDb::open_at(&db_path)` twice and asserts: - -```rust -assert_eq!(count_rows(&db_path, "lcm_raw_messages").await, 1); -assert_eq!(schema_version(&db_path).await, tokensave::sessions::lcm::LCM_SCHEMA_VERSION); -``` - -Run: `cargo test --test session_lcm_schema_test -- --nocapture` - -Expected: PASS. - -- [ ] **Step 7: Commit checkpoint** - -```bash -git add src/sessions/mod.rs src/sessions/lcm/mod.rs src/sessions/lcm/types.rs src/sessions/lcm/schema.rs src/global_db.rs tests/session_lcm_schema_test.rs -git commit -m "Add LCM session schema migrations" -``` - -## Task 2: Lossless Raw-Message Ingest Model - -**Files:** -- Create: `src/sessions/lcm/raw.rs` -- Create: `src/sessions/lcm/store.rs` -- Modify: `src/sessions/lcm/mod.rs` -- Modify: `src/sessions/lcm/types.rs` -- Modify: `src/global_db.rs` -- Modify: `src/sessions/source.rs` -- Modify: `tests/session_global_db_test.rs` -- Create: `tests/session_lcm_raw_test.rs` - -- [ ] **Step 1: Replace the capped-authoritative regression test with a lossless red test** - -In `tests/session_global_db_test.rs`, replace `upsert_session_message_truncates_oversized_text_deterministically` with: - -```rust -#[tokio::test] -async fn upsert_session_message_preserves_oversized_text_losslessly() { - let tmp = TempDir::new().unwrap(); - let db = open_isolated_db(&tmp).await; - let session = sample_session("cursor", "session-1", "project-a"); - db.upsert_session(&session).await; - - let oversized = format!("{}{}", "x".repeat(300_000), "::lossless-tail"); - let message = sample_message("cursor", "message-1", "session-1", &oversized); - assert!(db.upsert_session_message(&message).await); - - let compatibility = db - .get_session_message("cursor", "message-1") - .await - .expect("compatibility message should exist"); - assert!(compatibility.text.len() <= tokensave::sessions::lcm::MAX_DERIVED_TEXT_CHARS); - assert!(compatibility.text.contains("[derived snippet truncated by tokensave]")); - - let raw = db - .lcm_load_raw_message("cursor", "message-1") - .await - .expect("raw message should exist"); - assert_eq!(raw.content, oversized); - assert!(raw.content.ends_with("::lossless-tail")); - assert!(!raw.legacy_source); - assert!(!raw.legacy_truncated); -} -``` - -- [ ] **Step 2: Run the red test** - -Run: `cargo test --test session_global_db_test upsert_session_message_preserves_oversized_text_losslessly -- --nocapture` - -Expected: FAIL because `GlobalDb::upsert_session_message` still calls `capped_session_message_text` before storing the only authoritative text copy. - -- [ ] **Step 3: Add explicit derived-text helpers** - -Sketch: - -```rust -// src/sessions/lcm/types.rs -pub const MAX_DERIVED_TEXT_CHARS: usize = 64 * 1024; -pub const DERIVED_TRUNCATION_MARKER: &str = "\n[derived snippet truncated by tokensave]"; -``` - -```rust -// src/sessions/lcm/raw.rs -pub fn derived_text_for_index(raw: &str) -> String { - if raw.chars().count() <= MAX_DERIVED_TEXT_CHARS { - return raw.to_string(); - } - let mut out = raw.chars().take(MAX_DERIVED_TEXT_CHARS).collect::(); - out.push_str(DERIVED_TRUNCATION_MARKER); - out -} -``` - -- [ ] **Step 4: Add raw ingest API and make `upsert_session_message` write both layers** - -Sketch: - -```rust -impl GlobalDb { - pub async fn upsert_session_message(&self, message: &SessionMessageRecord) -> bool { - let Some(raw) = crate::sessions::lcm::raw::prepare_raw_message(message) else { - return false; - }; - if !crate::sessions::lcm::raw::upsert_raw_message(&self.conn, &raw).await { - return false; - } - let derived = crate::sessions::lcm::raw::derived_text_for_index(&message.text); - self.upsert_session_message_projection(message, &derived).await - } -} -``` - -Keep the existing `session_messages` table and FTS triggers as compatibility projections. Move the previous SQL body into a private `upsert_session_message_projection`. - -- [ ] **Step 5: Route transcript ingestion through the same write path** - -`src/sessions/source.rs` already calls `db.upsert_session_message(message)`. Keep that call so all providers inherit lossless behavior through one path. Add a test in `tests/session_lcm_raw_test.rs` using a fake `TranscriptSource` that emits a 300 KiB assistant message and assert the raw message tail survives. - -Run: `cargo test --test session_lcm_raw_test transcript_ingest_preserves_lossless_raw_content -- --nocapture` - -Expected before implementation: FAIL with missing fake-source helpers or missing raw content. Expected after implementation: PASS. - -- [ ] **Step 6: Prove search still uses bounded compatibility text** - -Add `search_uses_bounded_projection_but_load_recovers_raw`: - -```rust -let results = db.search_session_messages("cursor", Some("project-a"), "unique-search-token", 10).await; -assert_eq!(results.len(), 1); -assert!(results[0].message.text.len() <= tokensave::sessions::lcm::MAX_DERIVED_TEXT_CHARS + 64); -assert_eq!( - db.lcm_load_raw_message("cursor", "message-1").await.unwrap().content, - oversized -); -``` - -Run: `cargo test --test session_lcm_raw_test -- --nocapture` - -Expected: PASS. - -- [ ] **Step 7: Commit checkpoint** - -```bash -git add src/global_db.rs src/sessions/source.rs src/sessions/lcm/mod.rs src/sessions/lcm/types.rs src/sessions/lcm/raw.rs src/sessions/lcm/store.rs tests/session_global_db_test.rs tests/session_lcm_raw_test.rs -git commit -m "Make session raw ingest lossless" -``` - -## Task 3: External Payload Containment and Derived Index Separation - -**Files:** -- Create: `src/sessions/lcm/payload.rs` -- Create: `src/sessions/lcm/security.rs` -- Modify: `src/sessions/lcm/mod.rs` -- Modify: `src/sessions/lcm/types.rs` -- Modify: `src/sessions/lcm/schema.rs` -- Modify: `src/sessions/lcm/raw.rs` -- Create: `tests/session_lcm_payload_test.rs` -- Create: `tests/session_lcm_ingest_protection_test.rs` - -- [ ] **Step 1: Write failing externalization tests** - -Add `externalizes_large_tool_payload_with_recoverable_ref`: - -```rust -#[tokio::test] -async fn externalizes_large_tool_payload_with_recoverable_ref() { - let tmp = tempfile::TempDir::new().unwrap(); - let db = open_lcm_db(tmp.path()).await; - insert_session(&db, "cursor", "session-1").await; - - let payload = format!("tool output\n{}", "A".repeat(900_000)); - let message = raw_message("cursor", "tool-1", "session-1", "tool", &payload) - .with_kind("tool_result"); - db.lcm_store(tmp.path().join(".tokensave")) - .ingest_raw_message(&message) - .await - .unwrap(); - - let raw = db.lcm_load_raw_message("cursor", "tool-1").await.unwrap(); - assert_eq!(raw.storage_kind, LcmStorageKind::External); - assert!(raw.payload_ref.as_deref().unwrap().ends_with(".payload")); - assert_eq!( - db.lcm_expand_payload("cursor", "session-1", raw.payload_ref.as_deref().unwrap(), 0, payload.len()) - .await - .unwrap() - .content, - payload - ); -} -``` - -Run: `cargo test --test session_lcm_payload_test externalizes_large_tool_payload_with_recoverable_ref -- --nocapture` - -Expected: FAIL because payload externalization APIs do not exist. - -- [ ] **Step 2: Define payload metadata and storage root API** - -Sketch: - -```rust -#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] -pub struct LcmPayloadRef { - pub payload_ref: String, - pub provider: String, - pub session_id: String, - pub message_id: String, - pub kind: String, - pub content_hash: String, - pub byte_count: u64, - pub char_count: u64, - pub created_at: i64, -} -``` - -```rust -pub fn payload_dir(storage_root: &Path) -> PathBuf { - storage_root.join("lcm-payloads") -} -``` - -For project-local Cursor/Hermes installs, `storage_root` is project `.tokensave`. For non-local Hermes profile installs, `storage_root` is the selected Hermes profile root joined with `.tokensave`. - -- [ ] **Step 3: Add schema for payload refs** - -Add to `ensure_lcm_schema`: - -```sql -CREATE TABLE IF NOT EXISTS lcm_external_payloads ( - payload_ref TEXT PRIMARY KEY, - provider TEXT NOT NULL, - session_id TEXT NOT NULL, - message_id TEXT NOT NULL, - kind TEXT NOT NULL, - content_hash TEXT NOT NULL, - byte_count INTEGER NOT NULL, - char_count INTEGER NOT NULL, - created_at INTEGER NOT NULL DEFAULT (unixepoch()), - metadata_json TEXT, - UNIQUE(provider, message_id, payload_ref), - FOREIGN KEY(provider, session_id) REFERENCES sessions(provider, session_id) ON DELETE CASCADE -); -CREATE INDEX IF NOT EXISTS idx_lcm_external_payloads_owner - ON lcm_external_payloads(provider, session_id); -``` - -- [ ] **Step 4: Implement basename-only refs, containment, permissions, and hashes** - -Sketch: - -```rust -pub fn validate_payload_ref(payload_ref: &str) -> Result<&str> { - let path = std::path::Path::new(payload_ref); - let is_basename = path.components().count() == 1 - && !payload_ref.contains('/') - && !payload_ref.contains('\\') - && payload_ref != "." - && payload_ref != ".."; - if is_basename { Ok(payload_ref) } else { Err(LcmError::InvalidPayloadRef) } -} - -pub async fn write_external_payload( - root: &Path, - provider: &str, - session_id: &str, - message_id: &str, - content: &str, -) -> Result { - let hash = sha256_hex(content.as_bytes()); - let payload_ref = format!("{provider}-{session_id}-{message_id}-{hash}.payload"); - validate_payload_ref(&payload_ref)?; - let dir = payload_dir(root); - create_private_dir(&dir)?; - let path = dir.join(&payload_ref); - ensure_contained(&dir, &path)?; - write_private_file(&path, content.as_bytes())?; - Ok(LcmPayloadRef { payload_ref, provider: provider.to_string(), session_id: session_id.to_string(), message_id: message_id.to_string(), kind: "message".to_string(), content_hash: hash, byte_count: content.len() as u64, char_count: content.chars().count() as u64, created_at: unixepoch(), metadata_json: None }) -} -``` - -- [ ] **Step 5: Add protection tests for unsafe refs and cross-session expansion** - -Tests: - -```rust -#[test] -fn rejects_payload_ref_path_traversal() { - for bad in ["../secret", "/tmp/secret", "nested/file", r"nested\file", ".", ".."] { - assert!(tokensave::sessions::lcm::payload::validate_payload_ref(bad).is_err()); - } -} -``` - -```rust -#[tokio::test] -async fn denies_cross_session_payload_expansion() { - let ref_for_a = insert_external_payload(&db, "cursor", "session-a", "message-a", "secret").await; - let denied = db.lcm_expand_payload("cursor", "session-b", &ref_for_a, 0, 100).await; - assert!(matches!(denied, Err(LcmError::PayloadNotOwnedBySession))); -} -``` - -Run: `cargo test --test session_lcm_payload_test -- --nocapture` - -Expected: PASS after containment implementation. - -- [ ] **Step 6: Add data URI/base64/large-output classification tests** - -`tests/session_lcm_ingest_protection_test.rs`: - -```rust -#[test] -fn classifies_data_uri_and_long_base64_for_externalization() { - let data_uri = format!("data:image/png;base64,{}", "A".repeat(20_000)); - assert!(should_externalize("assistant", Some("tool_result"), &data_uri)); - - let base64_run = "Q".repeat(80_000); - assert!(should_externalize("assistant", Some("message"), &base64_run)); - - assert!(!should_externalize("assistant", Some("message"), "short useful text")); -} -``` - -Run: `cargo test --test session_lcm_ingest_protection_test -- --nocapture` - -Expected: PASS after `src/sessions/lcm/security.rs` classifies large/binary-ish content. - -- [ ] **Step 7: Commit checkpoint** - -```bash -git add src/sessions/lcm/mod.rs src/sessions/lcm/types.rs src/sessions/lcm/schema.rs src/sessions/lcm/raw.rs src/sessions/lcm/payload.rs src/sessions/lcm/security.rs tests/session_lcm_payload_test.rs tests/session_lcm_ingest_protection_test.rs -git commit -m "Add LCM payload containment" -``` - -## Task 4: Summary DAG Tables, Types, and Lineage Expansion - -**Files:** -- Create: `src/sessions/lcm/dag.rs` -- Modify: `src/sessions/lcm/mod.rs` -- Modify: `src/sessions/lcm/types.rs` -- Modify: `src/sessions/lcm/schema.rs` -- Create: `tests/session_lcm_dag_test.rs` - -- [ ] **Step 1: Write failing DAG persistence test** - -```rust -#[tokio::test] -async fn summary_node_preserves_source_lineage_and_expands_sources() { - let tmp = tempfile::TempDir::new().unwrap(); - let db = open_lcm_db(tmp.path()).await; - insert_raw_messages(&db, "cursor", "session-1", ["alpha", "beta", "gamma"]).await; - - let node = db.lcm_insert_summary_node(LcmSummaryNodeDraft { - provider: "cursor".into(), - conversation_id: "conversation-1".into(), - session_id: "session-1".into(), - depth: 0, - summary_text: "alpha through gamma".into(), - source_refs: vec![ - LcmSourceRef::RawMessage { store_id: 1 }, - LcmSourceRef::RawMessage { store_id: 2 }, - LcmSourceRef::RawMessage { store_id: 3 }, - ], - source_token_count: 30, - summary_token_count: 4, - source_time_start: Some(1_715_000_000), - source_time_end: Some(1_715_000_030), - expand_hint: Some("3 raw messages".into()), - metadata_json: None, - }).await.unwrap(); - - let expanded = db.lcm_expand_summary_node("cursor", "session-1", &node.node_id).await.unwrap(); - assert_eq!(expanded.sources.len(), 3); - assert_eq!(expanded.sources[0].content, "alpha"); - assert_eq!(expanded.summary.summary_text, "alpha through gamma"); -} -``` - -Run: `cargo test --test session_lcm_dag_test summary_node_preserves_source_lineage_and_expands_sources -- --nocapture` - -Expected: FAIL with missing `LcmSummaryNodeDraft`, `LcmSourceRef`, and DAG APIs. - -- [ ] **Step 2: Add DAG schema** - -Add: - -```sql -CREATE TABLE IF NOT EXISTS lcm_summary_nodes ( - node_id TEXT PRIMARY KEY, - provider TEXT NOT NULL, - conversation_id TEXT NOT NULL, - session_id TEXT NOT NULL, - depth INTEGER NOT NULL, - summary_text TEXT NOT NULL, - summary_hash TEXT NOT NULL, - summary_token_count INTEGER NOT NULL, - source_token_count INTEGER NOT NULL, - source_time_start INTEGER, - source_time_end INTEGER, - expand_hint TEXT, - metadata_json TEXT, - created_at INTEGER NOT NULL DEFAULT (unixepoch()) -); -CREATE TABLE IF NOT EXISTS lcm_summary_sources ( - node_id TEXT NOT NULL, - source_kind TEXT NOT NULL CHECK(source_kind IN ('raw_message', 'summary_node')), - source_id TEXT NOT NULL, - ordinal INTEGER NOT NULL, - PRIMARY KEY(node_id, ordinal), - FOREIGN KEY(node_id) REFERENCES lcm_summary_nodes(node_id) ON DELETE CASCADE -); -CREATE INDEX IF NOT EXISTS idx_lcm_summary_nodes_session - ON lcm_summary_nodes(provider, session_id, depth, created_at); -CREATE VIRTUAL TABLE IF NOT EXISTS lcm_summary_nodes_fts USING fts5( - summary_text, expand_hint, metadata_json, - content='lcm_summary_nodes', - content_rowid='rowid' -); -``` - -- [ ] **Step 3: Define stable DAG types** - -```rust -#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] -#[serde(tag = "kind", rename_all = "snake_case")] -pub enum LcmSourceRef { - RawMessage { store_id: i64 }, - SummaryNode { node_id: String }, -} - -#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] -pub struct LcmSummaryNode { - pub node_id: String, - pub provider: String, - pub conversation_id: String, - pub session_id: String, - pub depth: i64, - pub summary_text: String, - pub source_refs: Vec, - pub summary_token_count: i64, - pub source_token_count: i64, - pub source_time_start: Option, - pub source_time_end: Option, - pub expand_hint: Option, - pub metadata_json: Option, -} -``` - -- [ ] **Step 4: Implement deterministic node IDs and source expansion** - -Use a deterministic ID to make tests stable: - -```rust -pub fn summary_node_id(provider: &str, session_id: &str, depth: i64, source_refs: &[LcmSourceRef], summary_text: &str) -> String { - let input = serde_json::json!({ - "provider": provider, - "session_id": session_id, - "depth": depth, - "source_refs": source_refs, - "summary_hash": sha256_hex(summary_text.as_bytes()), - }); - format!("sum_{}", sha256_hex(input.to_string().as_bytes())) -} -``` - -Expansion must check `(provider, session_id)` ownership before returning raw messages or child summary nodes. - -- [ ] **Step 5: Add restart recovery test** - -Create `summary_dag_survives_reopen` that inserts a node, drops `GlobalDb`, reopens the same `sessions.db`, and expands the node. - -Run: `cargo test --test session_lcm_dag_test -- --nocapture` - -Expected: PASS. - -- [ ] **Step 6: Commit checkpoint** - -```bash -git add src/sessions/lcm/mod.rs src/sessions/lcm/types.rs src/sessions/lcm/schema.rs src/sessions/lcm/dag.rs tests/session_lcm_dag_test.rs -git commit -m "Add LCM summary DAG storage" -``` - -## Task 5: Search, Load, Expand, Describe, and Status Rust APIs - -**Files:** -- Create: `src/sessions/lcm/query.rs` -- Modify: `src/sessions/lcm/mod.rs` -- Modify: `src/sessions/lcm/types.rs` -- Modify: `src/sessions/lcm/raw.rs` -- Modify: `src/sessions/lcm/dag.rs` -- Modify: `src/sessions/lcm/payload.rs` -- Create: `tests/session_lcm_query_test.rs` - -- [ ] **Step 1: Write failing query API tests** - -```rust -#[tokio::test] -async fn load_session_returns_ordered_raw_pages_with_stable_cursor() { - let tmp = tempfile::TempDir::new().unwrap(); - let db = open_lcm_db(tmp.path()).await; - insert_raw_messages(&db, "cursor", "session-1", ["one", "two", "three"]).await; - - let page = db.lcm_load_session(LcmLoadSessionRequest { - provider: "cursor".into(), - session_id: "session-1".into(), - after_store_id: None, - limit: 2, - role: None, - start_time: None, - end_time: None, - content_slice: None, - }).await.unwrap(); - - assert_eq!(page.messages.iter().map(|m| m.content.as_str()).collect::>(), ["one", "two"]); - assert_eq!(page.next_cursor.as_deref(), Some("2")); - - let second = db.lcm_load_session(LcmLoadSessionRequest { - after_store_id: Some(2), - ..page.request_for_next() - }).await.unwrap(); - assert_eq!(second.messages[0].content, "three"); - assert!(second.next_cursor.is_none()); -} -``` - -Run: `cargo test --test session_lcm_query_test load_session_returns_ordered_raw_pages_with_stable_cursor -- --nocapture` - -Expected: FAIL because query request/response types do not exist. - -- [ ] **Step 2: Add query request/response types** - -Sketch: - -```rust -pub struct LcmLoadSessionRequest { - pub provider: String, - pub session_id: String, - pub after_store_id: Option, - pub limit: usize, - pub role: Option, - pub start_time: Option, - pub end_time: Option, - pub content_slice: Option, -} - -pub struct LcmGrepRequest { - pub provider: String, - pub query: String, - pub scope: LcmScope, - pub session_id: Option, - pub include_summaries: bool, - pub limit: usize, -} - -pub enum LcmScope { - Current, - Session, - All, -} -``` - -- [ ] **Step 3: Implement load and expand with pagination** - -Rules: - -- `limit` clamps to `1..=100`. -- `content_slice` returns bounded `content` plus `content_range`, never silently discards raw content. -- External payload expansion uses `LcmPayloadRef` ownership checks from Task 3. -- Summary expansion uses DAG source ownership checks from Task 4. - -Run: `cargo test --test session_lcm_query_test load_session_returns_ordered_raw_pages_with_stable_cursor -- --nocapture` - -Expected: PASS. - -- [ ] **Step 4: Write and implement combined grep/status/describe tests** - -Tests: - -```rust -#[tokio::test] -async fn grep_searches_raw_snippets_and_summary_nodes() { - let hits = db.lcm_grep(LcmGrepRequest { - provider: "cursor".into(), - query: "billing migration".into(), - scope: LcmScope::Session, - session_id: Some("session-1".into()), - include_summaries: true, - limit: 10, - }).await.unwrap(); - assert!(hits.iter().any(|hit| hit.kind == "raw_message")); - assert!(hits.iter().any(|hit| hit.kind == "summary_node")); - assert!(hits.iter().all(|hit| hit.snippet.len() <= MAX_DERIVED_TEXT_CHARS + 64)); -} -``` - -```rust -#[tokio::test] -async fn status_reports_schema_frontier_payload_and_debt_counts() { - let status = db.lcm_status("cursor", Some("session-1")).await.unwrap(); - assert_eq!(status.schema_version, LCM_SCHEMA_VERSION); - assert_eq!(status.raw_message_count, 3); - assert_eq!(status.summary_node_count, 1); - assert_eq!(status.missing_payload_count, 0); - assert_eq!(status.maintenance_debt_count, 0); -} -``` - -Run: `cargo test --test session_lcm_query_test -- --nocapture` - -Expected: PASS. - -- [ ] **Step 5: Commit checkpoint** - -```bash -git add src/sessions/lcm/mod.rs src/sessions/lcm/types.rs src/sessions/lcm/raw.rs src/sessions/lcm/dag.rs src/sessions/lcm/payload.rs src/sessions/lcm/query.rs tests/session_lcm_query_test.rs -git commit -m "Add LCM query APIs" -``` - -## Task 6: MCP Tool Definitions, Handlers, and Message Search Compatibility - -**Files:** -- Modify: `src/mcp/tools/definitions.rs` -- Modify: `src/mcp/tools/handlers/session.rs` -- Modify: `src/mcp/tools/handlers/mod.rs` -- Modify: `tests/mcp_handler_test.rs` -- Modify: `tests/session_global_db_test.rs` - -- [ ] **Step 1: Write failing tool-definition tests** - -In `tests/mcp_handler_test.rs`: - -```rust -#[test] -fn lcm_tool_schemas_are_registered_with_stable_names() { - let tools = get_tool_definitions(); - let names = tools.iter().map(|tool| tool.name.as_str()).collect::>(); - - for expected in [ - "tokensave_lcm_status", - "tokensave_lcm_load_session", - "tokensave_lcm_grep", - "tokensave_lcm_describe", - "tokensave_lcm_expand", - "tokensave_lcm_expand_query", - "tokensave_lcm_preflight", - "tokensave_lcm_compress", - ] { - assert!(names.contains(expected), "missing {expected}"); - } -} -``` - -Run: `cargo test --test mcp_handler_test lcm_tool_schemas_are_registered_with_stable_names -- --nocapture` - -Expected: FAIL because definitions are not registered. - -- [ ] **Step 2: Add additive definitions with precise JSON schemas** - -Sketch for one tool: - -```rust -fn def_lcm_load_session() -> ToolDefinition { - def( - "tokensave_lcm_load_session", - "LCM Load Session", - "Load ordered lossless raw session messages with pagination and bounded response slicing.", - json!({ - "type": "object", - "properties": { - "provider": {"type": "string", "description": "Provider id, default cursor."}, - "session_id": {"type": "string", "description": "Provider-local session id."}, - "after_store_id": {"type": "number", "description": "Return rows after this raw store id."}, - "limit": {"type": "number", "description": "Maximum rows, clamped to 100."}, - "role": {"type": "string", "description": "Optional role filter."}, - "content_offset": {"type": "number", "description": "Character offset for content slice."}, - "content_limit": {"type": "number", "description": "Maximum characters returned per message."} - }, - "required": ["session_id"] - }), - ) -} -``` - -Use `def_rw` for `tokensave_lcm_preflight` and `tokensave_lcm_compress` because they ingest or mutate lifecycle state. - -- [ ] **Step 3: Add handler dispatch and JSON output contracts** - -Sketch: - -```rust -pub(super) async fn handle_lcm_load_session(cg: &TokenSave, args: Value) -> Result { - let provider = string_arg(&args, "provider").unwrap_or("cursor"); - let session_id = required_string_arg(&args, "session_id")?; - let db = open_project_session_db_or_unavailable(cg.project_root()).await?; - let page = db.lcm_load_session(LcmLoadSessionRequest::from_args(provider, session_id, &args)?).await?; - Ok(tool_json(&json!({ - "status": "ok", - "provider": provider, - "session_id": session_id, - "messages": page.messages, - "next_cursor": page.next_cursor, - }))) -} -``` - -Keep `truncate_response` at the outer MCP rendering layer, not in raw storage. - -- [ ] **Step 4: Preserve `tokensave_message_search` behavior** - -Add a compatibility test that reuses current assertions: - -```rust -#[tokio::test] -async fn message_search_preserves_provider_project_parent_scope_shape_after_lcm() { - let (cg, _dir) = setup_project().await; - seed_lcm_and_projection_rows(cg.project_root()).await; - - let result = handle_tool_call( - &cg, - "tokensave_message_search", - json!({ - "query": "orchard dispatch", - "provider": "cursor", - "project_key": cg.project_root().to_string_lossy(), - "scope": "subagents_only", - "parent_session_id": "parent", - "limit": 10 - }), - None, - None, - ).await.unwrap(); - - let payload: serde_json::Value = serde_json::from_str(extract_text(&result.value)).unwrap(); - assert_eq!(payload["status"], "ok"); - assert_eq!(payload["scope"], "subagents_only"); - assert_eq!(payload["results"].as_array().unwrap().len(), 1); - assert!(payload["results"][0]["message"].get("text").is_some()); -} -``` - -Run: `cargo test --test mcp_handler_test message_search_preserves_provider_project_parent_scope_shape_after_lcm -- --nocapture` - -Expected: PASS after handler compatibility is wired. - -- [ ] **Step 5: Add CLI bridge smoke test for `tokensave tool ... --json --args`** - -In `tests/mcp_handler_test.rs` or a new CLI integration test: - -```rust -let output = std::process::Command::new(env!("CARGO_BIN_EXE_tokensave")) - .current_dir(cg.project_root()) - .args([ - "tool", - "tokensave_lcm_status", - "--json", - "--args", - r#"{"provider":"cursor"}"#, - ]) - .output() - .unwrap(); -assert!(output.status.success()); -let json: serde_json::Value = serde_json::from_slice(&output.stdout).unwrap(); -assert_eq!(json["content"][0]["type"], "text"); -``` - -Run: `cargo test --test mcp_handler_test lcm_tool_schemas_are_registered_with_stable_names message_search_preserves_provider_project_parent_scope_shape_after_lcm -- --nocapture` - -Expected: PASS. - -- [ ] **Step 6: Commit checkpoint** - -```bash -git add src/mcp/tools/definitions.rs src/mcp/tools/handlers/session.rs src/mcp/tools/handlers/mod.rs tests/mcp_handler_test.rs tests/session_global_db_test.rs -git commit -m "Expose LCM session tools" -``` - -## Task 7: Hermes Context Engine Registration and Storage Locality - -**Files:** -- Modify: `src/agents/hermes.rs` -- Modify: `tests/agent_test.rs` -- Create: `tests/hermes_lcm_bridge_test.rs` - -- [ ] **Step 1: Write failing generated-plugin registration test** - -In `tests/agent_test.rs`: - -```rust -#[test] -fn test_hermes_generated_python_registers_lcm_context_engine() { - let home = TempDir::new().unwrap(); - HermesIntegration.install(&make_install_ctx(home.path())).unwrap(); - let init_py = std::fs::read_to_string(home.path().join(".hermes/plugins/tokensave/__init__.py")).unwrap(); - - assert!(init_py.contains("class TokenSaveContextEngine")); - assert!(init_py.contains("ctx.register_context_engine")); - assert!(init_py.contains("tools.call_tokensave_tool(\"tokensave_lcm_preflight\"")); - assert!(init_py.contains("tools.call_tokensave_tool(\"tokensave_lcm_compress\"")); -} -``` - -Run: `cargo test --test agent_test test_hermes_generated_python_registers_lcm_context_engine -- --nocapture` - -Expected: FAIL because generated Python does not register a context engine yet. - -- [ ] **Step 2: Generate explicit storage locality metadata** - -Add helper output in `plugin_init()`: - -```python -def _storage_args(project_root=None, hermes_home=None): - args = {} - if project_root: - args["storage_scope"] = "project_local" - args["project_root"] = str(project_root) - elif hermes_home: - args["storage_scope"] = "hermes_profile" - args["hermes_home"] = str(hermes_home) - else: - args["storage_scope"] = "hermes_profile" - return args -``` - -Local Hermes install (`tokensave install --local --agent hermes` without `--profile`) continues to write under `project/.hermes/plugins/tokensave` and instructs users to launch with `HERMES_HOME=project/.hermes`. The LCM storage args still point Rust at the project `.tokensave/sessions.db` when `project_root` is known. - -Non-local/profile Hermes install uses the active Hermes home/profile as the storage root; Rust resolves the session DB at `/.tokensave/sessions.db`. - -- [ ] **Step 3: Add generated `TokenSaveContextEngine` skeleton** - -Sketch: - -```python -class TokenSaveContextEngine: - def __init__(self): - self.hermes_home = None - self.project_root = None - self.active_session_id = None - - def initialize(self, session_id=None, hermes_home=None, project_root=None, **kwargs): - self.active_session_id = session_id - self.hermes_home = hermes_home - self.project_root = project_root or kwargs.get("cwd") - - def should_compress_preflight(self, messages, **kwargs): - args = _storage_args(self.project_root, self.hermes_home) - args.update({"session_id": self.active_session_id, "messages": messages}) - return json.loads(tools.call_tokensave_tool("tokensave_lcm_preflight", args)) - - def compress(self, messages, current_tokens=None, focus_topic=None, **kwargs): - args = _storage_args(self.project_root, self.hermes_home) - args.update({ - "session_id": self.active_session_id, - "messages": messages, - "current_tokens": current_tokens, - "focus_topic": focus_topic, - }) - return json.loads(tools.call_tokensave_tool("tokensave_lcm_compress", args)) -``` - -In `register(ctx)`, call `ctx.register_context_engine(TokenSaveContextEngine())` only when the method exists, matching the current optional registration style for commands and memory providers. - -- [ ] **Step 4: Add Python compile and fake context tests** - -In `tests/hermes_lcm_bridge_test.rs`, install the plugin into a temp home and run Python that imports `__init__.py`, supplies a fake `ctx`, and asserts `register_context_engine` receives a `TokenSaveContextEngine`. - -Run: `cargo test --test hermes_lcm_bridge_test generated_context_engine_registers_when_supported -- --nocapture` - -Expected: PASS after generated Python changes. - -- [ ] **Step 5: Commit checkpoint** - -```bash -git add src/agents/hermes.rs tests/agent_test.rs tests/hermes_lcm_bridge_test.rs -git commit -m "Register Hermes LCM context engine" -``` - -## Task 8: Python Bridge Calls and Deterministic No-Op/Fake Summarizer Tests - -**Files:** -- Modify: `src/agents/hermes.rs` -- Modify: `src/sessions/lcm/hermes.rs` -- Modify: `src/sessions/lcm/types.rs` -- Modify: `tests/hermes_lcm_bridge_test.rs` - -- [ ] **Step 1: Write failing bridge-call argument test** - -In `tests/hermes_lcm_bridge_test.rs`, generate plugin files and run a Python script that monkeypatches `tools.subprocess.run`: - -```python -calls = [] -def fake_run(argv, check, capture_output, text, timeout, shell): - calls.append(argv) - payload = {"content": [{"type": "text", "text": "{\"status\":\"ok\",\"should_compress\":false,\"messages\":[]}"}]} - return Result(0, json.dumps(payload), "") - -tools.subprocess.run = fake_run -engine = plugin.TokenSaveContextEngine() -engine.initialize(session_id="session-1", project_root="/tmp/project") -result = engine.should_compress_preflight([{"role": "user", "content": "hello"}]) - -assert result["status"] == "ok" -assert calls[0][0] == tools.TOKENSAVE_BIN -assert calls[0][1:4] == ["tool", "tokensave_lcm_preflight", "--json"] -assert "--args" in calls[0] -assert '"storage_scope": "project_local"' in calls[0][-1] -``` - -Run: `cargo test --test hermes_lcm_bridge_test context_engine_preflight_uses_tokensave_tool_json_args -- --nocapture` - -Expected: FAIL until the generated context engine uses `tools.call_tokensave_tool` with the correct names and storage args. - -- [ ] **Step 2: Normalize nested MCP JSON text in generated Python** - -Current `tools.call_tokensave_tool` returns the full JSON-RPC result string. Add a helper: - -```python -def call_tokensave_json(name: str, args: dict, **kwargs): - raw = call_tokensave_tool(name, args, **kwargs) - outer = json.loads(raw) - text = outer.get("content", [{}])[0].get("text", "{}") - return json.loads(text) -``` - -Use this helper inside `TokenSaveContextEngine`. - -- [ ] **Step 3: Add deterministic fake summarizer contract** - -Rust request/response sketch: - -```rust -#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] -pub struct LcmCompressionRequest { - pub provider: String, - pub session_id: String, - pub messages: Vec, - pub current_tokens: Option, - pub focus_topic: Option, - pub summarizer: LcmSummarizerMode, -} - -#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] -#[serde(tag = "mode", rename_all = "snake_case")] -pub enum LcmSummarizerMode { - Noop, - Fake { summary_text: String }, - HermesAuxiliary, -} -``` - -The generated Python uses `HermesAuxiliary` in production. Rust tests use `Noop` or `Fake`. - -- [ ] **Step 4: Add no-op preflight/compress tests** - -Tests: - -```rust -#[tokio::test] -async fn noop_summarizer_ingests_messages_without_summary_nodes() { - let response = db.lcm_compress(LcmCompressionRequest { - provider: "cursor".into(), - session_id: "session-1".into(), - messages: vec![json!({"role": "user", "content": "fresh"})], - current_tokens: Some(100), - focus_topic: None, - summarizer: LcmSummarizerMode::Noop, - }).await.unwrap(); - - assert_eq!(response.status, "ok"); - assert_eq!(response.summary_nodes_created, 0); - assert_eq!(db.lcm_load_session(raw_load("cursor", "session-1")).await.unwrap().messages.len(), 1); -} -``` - -Run: `cargo test --test hermes_lcm_bridge_test context_engine_preflight_uses_tokensave_tool_json_args -- --nocapture` - -Expected: PASS. - -- [ ] **Step 5: Commit checkpoint** - -```bash -git add src/agents/hermes.rs src/sessions/lcm/hermes.rs src/sessions/lcm/types.rs tests/hermes_lcm_bridge_test.rs -git commit -m "Add Hermes LCM bridge contracts" -``` - -## Task 9: Compression Lifecycle Primitives and Deterministic Summarizer Injection - -**Files:** -- Create: `src/sessions/lcm/compression.rs` -- Modify: `src/sessions/lcm/mod.rs` -- Modify: `src/sessions/lcm/types.rs` -- Modify: `src/sessions/lcm/schema.rs` -- Modify: `src/sessions/lcm/dag.rs` -- Modify: `src/sessions/lcm/query.rs` -- Create: `tests/session_lcm_compression_test.rs` - -- [ ] **Step 1: Write failing lifecycle schema tests** - -```rust -#[tokio::test] -async fn lifecycle_frontier_survives_reopen() { - let tmp = tempfile::TempDir::new().unwrap(); - let db_path = tmp.path().join(".tokensave/sessions.db"); - let db = GlobalDb::open_at(&db_path).await.unwrap(); - db.lcm_update_lifecycle(LcmLifecycleUpdate { - provider: "cursor".into(), - conversation_id: "conversation-1".into(), - current_session_id: "session-1".into(), - current_frontier_store_id: Some(42), - last_finalized_session_id: Some("session-0".into()), - last_finalized_frontier_store_id: Some(40), - maintenance_debt: vec![LcmMaintenanceDebt::RawBacklog { from_store_id: 41, to_store_id: 42 }], - }).await.unwrap(); - drop(db); - - let reopened = GlobalDb::open_at(&db_path).await.unwrap(); - let state = reopened.lcm_lifecycle_state("cursor", "conversation-1").await.unwrap(); - assert_eq!(state.current_frontier_store_id, Some(42)); - assert_eq!(state.last_finalized_session_id.as_deref(), Some("session-0")); - assert_eq!(state.maintenance_debt.len(), 1); -} -``` - -Run: `cargo test --test session_lcm_compression_test lifecycle_frontier_survives_reopen -- --nocapture` - -Expected: FAIL with missing lifecycle table/types. - -- [ ] **Step 2: Add lifecycle/frontier/debt schema** - -```sql -CREATE TABLE IF NOT EXISTS lcm_lifecycle_state ( - provider TEXT NOT NULL, - conversation_id TEXT NOT NULL, - current_session_id TEXT NOT NULL, - last_finalized_session_id TEXT, - current_frontier_store_id INTEGER, - last_finalized_frontier_store_id INTEGER, - rollover_at INTEGER, - reset_at INTEGER, - maintenance_at INTEGER, - updated_at INTEGER NOT NULL DEFAULT (unixepoch()), - PRIMARY KEY(provider, conversation_id) -); -CREATE TABLE IF NOT EXISTS lcm_maintenance_debt ( - provider TEXT NOT NULL, - conversation_id TEXT NOT NULL, - debt_id TEXT NOT NULL, - debt_kind TEXT NOT NULL, - from_store_id INTEGER, - to_store_id INTEGER, - metadata_json TEXT, - created_at INTEGER NOT NULL DEFAULT (unixepoch()), - PRIMARY KEY(provider, conversation_id, debt_id), - FOREIGN KEY(provider, conversation_id) - REFERENCES lcm_lifecycle_state(provider, conversation_id) ON DELETE CASCADE -); -``` - -- [ ] **Step 3: Add fake summarizer trait and deterministic compression API** - -Sketch: - -```rust -#[async_trait::async_trait] -pub trait LcmSummarizer: Send + Sync { - async fn summarize(&self, request: LcmSummarizeRequest) -> Result; -} - -pub struct FakeSummarizer { - pub summary_text: String, -} - -#[async_trait::async_trait] -impl LcmSummarizer for FakeSummarizer { - async fn summarize(&self, request: LcmSummarizeRequest) -> Result { - Ok(LcmSummarizeOutput { - summary_text: self.summary_text.clone(), - source_token_count: request.source_token_count, - summary_token_count: estimate_tokens(&self.summary_text), - }) - } -} -``` - -If the repo avoids `async-trait`, define the trait as synchronous and keep Hermes auxiliary calls in generated Python. The Rust fake summarizer is enough for deterministic compression tests. - -- [ ] **Step 4: Implement preflight-ingests behavior** - -Test: - -```rust -#[tokio::test] -async fn preflight_can_request_compression_when_ingest_protection_changes_replay() { - let response = db.lcm_preflight(LcmPreflightRequest { - provider: "cursor".into(), - session_id: "session-1".into(), - messages: vec![json!({"role": "assistant", "content": format!("data:image/png;base64,{}", "A".repeat(100_000))})], - current_tokens: Some(100), - }).await.unwrap(); - - assert!(response.should_compress); - assert_eq!(response.reason, "ingest_protection_changed_replay"); - assert!(response.replay_messages[0]["content"].as_str().unwrap().contains("[externalized payload")); -} -``` - -- [ ] **Step 5: Implement frontier advance, fresh tail preservation, and DAG node creation** - -Test: - -```rust -#[tokio::test] -async fn fake_summarizer_compacts_backlog_and_preserves_fresh_tail() { - insert_raw_messages(&db, "cursor", "session-1", ["old-1", "old-2", "fresh-1", "fresh-2"]).await; - let response = db.lcm_compress_with_summarizer( - compress_request("cursor", "session-1").fresh_tail_count(2), - &FakeSummarizer { summary_text: "old summary".into() }, - ).await.unwrap(); - - assert_eq!(response.summary_nodes_created, 1); - assert_eq!(response.replay_messages.last().unwrap()["content"], "fresh-2"); - assert_eq!(response.frontier.current_frontier_store_id, Some(2)); -} -``` - -Run: `cargo test --test session_lcm_compression_test -- --nocapture` - -Expected: PASS. - -- [ ] **Step 6: Commit checkpoint** - -```bash -git add src/sessions/lcm/mod.rs src/sessions/lcm/types.rs src/sessions/lcm/schema.rs src/sessions/lcm/dag.rs src/sessions/lcm/query.rs src/sessions/lcm/compression.rs tests/session_lcm_compression_test.rs -git commit -m "Add deterministic LCM compression lifecycle" -``` - -## Task 10: Hermes Auxiliary LLM Bridge, Reasoning Stripping, and Fallbacks - -**Files:** -- Modify: `src/agents/hermes.rs` -- Modify: `src/sessions/lcm/hermes.rs` -- Modify: `src/sessions/lcm/types.rs` -- Modify: `tests/hermes_lcm_bridge_test.rs` - -- [ ] **Step 1: Write failing reasoning-strip test for generated Python** - -Python fixture in `tests/hermes_lcm_bridge_test.rs`: - -```python -class Aux: - def __init__(self): - self.calls = [] - def call_llm(self, **kwargs): - self.calls.append(kwargs) - return "hidden chain\nUseful compact summary" - -agent = type("Agent", (), {"auxiliary_client": Aux()})() -engine = plugin.TokenSaveContextEngine() -engine.initialize(session_id="session-1", project_root="/tmp/project", agent=agent) -summary = engine._call_auxiliary_summary("Summarize", [{"role": "user", "content": "raw"}]) - -assert summary["status"] == "ok" -assert summary["text"] == "Useful compact summary" -assert agent.auxiliary_client.calls[0]["task"] == "compression" -``` - -Run: `cargo test --test hermes_lcm_bridge_test auxiliary_summary_strips_reasoning_tags -- --nocapture` - -Expected: FAIL until auxiliary bridge helper exists. - -- [ ] **Step 2: Add generated Python reasoning stripper** - -Sketch: - -```python -REASONING_TAGS = ["think", "thinking", "reasoning", "thought", "REASONING_SCRATCHPAD"] - -def _strip_reasoning(text: str) -> str: - output = text or "" - for tag in REASONING_TAGS: - output = re.sub(fr"<{tag}>.*?", "", output, flags=re.DOTALL | re.IGNORECASE) - return output.strip() -``` - -- [ ] **Step 3: Add auxiliary summary route chain and cooldown state** - -Keep state process-local in Python: - -```python -class TokenSaveContextEngine: - def __init__(self): - self._route_failures = {} - self._cooldown_until = {} - - def _call_auxiliary_summary(self, prompt, messages, **kwargs): - routes = kwargs.get("routes") or [{"model": kwargs.get("model"), "temperature": 0.1}] - for route in routes: - key = route.get("model") or "default" - if self._cooldown_until.get(key, 0) > time.time(): - continue - try: - text = self.agent.auxiliary_client.call_llm( - task="compression", - messages=[{"role": "system", "content": prompt}, *messages], - temperature=route.get("temperature", 0.1), - max_tokens=route.get("max_tokens", 2048), - timeout=route.get("timeout", 60), - model=route.get("model"), - ) - stripped = _strip_reasoning(str(text)) - if stripped: - return {"status": "ok", "text": stripped, "route": key} - except Exception as exc: - self._route_failures[key] = self._route_failures.get(key, 0) + 1 - self._cooldown_until[key] = time.time() + min(300, 2 ** self._route_failures[key]) - return {"status": "fallback", "text": _deterministic_truncation(messages)} -``` - -- [ ] **Step 4: Pass auxiliary summaries back to Rust** - -`compress()` flow in generated Python: - -1. Call `tokensave_lcm_compress` with `summarizer.mode = "request_auxiliary"`. -2. If Rust returns `status = "needs_summary"` with a prompt and source messages, call `_call_auxiliary_summary`. -3. Call `tokensave_lcm_compress` again with `summarizer.mode = "provided"` and `summary_text`. -4. Return the final replay messages and frontier/status JSON to Hermes. - -The Rust contract: - -```rust -#[serde(tag = "mode", rename_all = "snake_case")] -pub enum LcmSummarizerMode { - Noop, - Fake { summary_text: String }, - RequestAuxiliary, - Provided { summary_text: String, route: Option }, -} -``` - -- [ ] **Step 5: Add fallback behavior tests** - -Tests: - -```python -class FailingAux: - def call_llm(self, **kwargs): - raise RuntimeError("route unavailable") - -engine.initialize(session_id="session-1", project_root="/tmp/project", agent=type("Agent", (), {"auxiliary_client": FailingAux()})()) -summary = engine._call_auxiliary_summary("Summarize", [{"role": "user", "content": "A" * 10_000}]) -assert summary["status"] == "fallback" -assert len(summary["text"]) < 10_000 -assert summary["text"].endswith("[deterministic compression fallback]") -``` - -Run: `cargo test --test hermes_lcm_bridge_test -- --nocapture` - -Expected: PASS. - -- [ ] **Step 6: Commit checkpoint** - -```bash -git add src/agents/hermes.rs src/sessions/lcm/hermes.rs src/sessions/lcm/types.rs tests/hermes_lcm_bridge_test.rs -git commit -m "Integrate Hermes auxiliary LCM summaries" -``` - -## Task 11: Ingest Protection, Diagnostics, Doctor, and Regression Tests - -**Files:** -- Modify: `src/sessions/lcm/security.rs` -- Modify: `src/sessions/lcm/query.rs` -- Modify: `src/mcp/tools/definitions.rs` -- Modify: `src/mcp/tools/handlers/session.rs` -- Modify: `src/agents/mod.rs` -- Modify: `src/agents/hermes.rs` -- Modify: `tests/session_lcm_ingest_protection_test.rs` -- Modify: `tests/session_lcm_payload_test.rs` -- Modify: `tests/agent_test.rs` -- Modify: `tests/mcp_handler_test.rs` -- Modify: `tests/session_lcm_query_test.rs` - -- [ ] **Step 1: Write failing doctor/status integrity tests** - -```rust -#[tokio::test] -async fn lcm_status_reports_missing_and_unreferenced_payloads_without_previewing_content() { - let tmp = tempfile::TempDir::new().unwrap(); - let db = open_lcm_db(tmp.path()).await; - let secret = "SUPER_SECRET_PAYLOAD"; - let payload_ref = insert_external_payload(&db, "cursor", "session-1", "message-1", secret).await; - std::fs::remove_file(payload_path(tmp.path(), &payload_ref)).unwrap(); - std::fs::write(payload_dir(tmp.path()).join("orphan.payload"), "orphan secret").unwrap(); - - let status = db.lcm_status("cursor", Some("session-1")).await.unwrap(); - assert_eq!(status.missing_payload_count, 1); - assert_eq!(status.unreferenced_payload_count, 1); - let rendered = serde_json::to_string(&status).unwrap(); - assert!(!rendered.contains(secret)); - assert!(!rendered.contains("orphan secret")); -} -``` - -Run: `cargo test --test session_lcm_payload_test lcm_status_reports_missing_and_unreferenced_payloads_without_previewing_content -- --nocapture` - -Expected: FAIL until integrity scan fields exist. - -- [ ] **Step 2: Add diagnostics JSON to `tokensave_lcm_status`** - -Status output sketch: - -```json -{ - "status": "ok", - "schema_version": 1, - "storage_scope": "project_local", - "raw_message_count": 12, - "summary_node_count": 2, - "payload": { - "externalized_count": 1, - "missing_count": 0, - "unreferenced_count": 0, - "root_contained": true - }, - "lifecycle": { - "current_session_id": "session-1", - "current_frontier_store_id": 10, - "maintenance_debt_count": 0 - }, - "redaction": { - "enabled": false, - "lossy_records": 0 - } -} -``` - -Do not include payload content in diagnostics. - -- [ ] **Step 3: Add regression scan against authoritative caps** - -Test: - -```rust -#[test] -fn no_authoritative_session_write_uses_legacy_text_cap() { - let source = std::fs::read_to_string("src/global_db.rs").unwrap(); - assert!(!source.contains("MAX_SESSION_MESSAGE_TEXT_BYTES")); - assert!(!source.contains("SESSION_MESSAGE_TRUNCATION_MARKER")); - - let lcm_raw = std::fs::read_to_string("src/sessions/lcm/raw.rs").unwrap(); - assert!(lcm_raw.contains("MAX_DERIVED_TEXT_CHARS")); - assert!(lcm_raw.contains("derived_text_for_index")); -} -``` - -Run: `cargo test --test session_lcm_ingest_protection_test no_authoritative_session_write_uses_legacy_text_cap -- --nocapture` - -Expected: PASS after old authoritative cap symbols are removed or renamed to derived-only constants. - -- [ ] **Step 4: Add separation regressions for codegraph and fact memory** - -In `tests/session_lcm_query_test.rs`: - -```rust -#[test] -fn lcm_modules_do_not_depend_on_context_builder_or_memory_fact_store() { - for path in [ - "src/sessions/lcm/raw.rs", - "src/sessions/lcm/dag.rs", - "src/sessions/lcm/query.rs", - "src/sessions/lcm/compression.rs", - ] { - let source = std::fs::read_to_string(path).unwrap(); - assert!(!source.contains("ContextBuilder")); - assert!(!source.contains("MemoryCategory")); - assert!(!source.contains("memory_facts")); - } -} -``` - -Run: `cargo test --test session_lcm_query_test lcm_modules_do_not_depend_on_context_builder_or_memory_fact_store -- --nocapture` - -Expected: PASS. - -- [ ] **Step 5: Extend Hermes doctor tests** - -In `tests/agent_test.rs`, assert Hermes healthcheck recognizes generated LCM plugin files and local/profile storage hints: - -```rust -assert!(init_py.contains("TokenSaveContextEngine")); -assert!(init_py.contains("storage_scope")); -assert!(manifest.contains("tokensave_lcm_status")); -assert!(manifest.contains("tokensave_lcm_compress")); -``` - -Run: `cargo test --test agent_test hermes -- --nocapture` - -Expected: PASS. - -- [ ] **Step 6: Run focused regression suite** - -Run: - -```bash -cargo test --test session_lcm_schema_test -- --nocapture -cargo test --test session_lcm_raw_test -- --nocapture -cargo test --test session_lcm_payload_test -- --nocapture -cargo test --test session_lcm_dag_test -- --nocapture -cargo test --test session_lcm_query_test -- --nocapture -cargo test --test session_lcm_compression_test -- --nocapture -cargo test --test session_lcm_ingest_protection_test -- --nocapture -cargo test --test hermes_lcm_bridge_test -- --nocapture -cargo test --test session_global_db_test -- --nocapture -cargo test --test mcp_handler_test -- --nocapture -cargo test --test agent_test hermes -- --nocapture -``` - -Expected: all commands PASS. - -- [ ] **Step 7: Commit checkpoint** - -```bash -git add src/sessions/lcm/security.rs src/sessions/lcm/query.rs src/mcp/tools/definitions.rs src/mcp/tools/handlers/session.rs src/agents/mod.rs src/agents/hermes.rs tests/session_lcm_ingest_protection_test.rs tests/session_lcm_payload_test.rs tests/agent_test.rs tests/mcp_handler_test.rs tests/session_lcm_query_test.rs -git commit -m "Add LCM diagnostics and regressions" -``` - -## Final Verification - -Run these after all implementation tasks are complete: - -```bash -cargo fmt --all -- --check -cargo clippy --all-targets -cargo test -``` - -Expected: - -- `cargo fmt --all -- --check` exits 0 with no formatting drift. -- `cargo clippy --all-targets` exits 0 with no new warnings. -- `cargo test` exits 0 across the full Rust test suite. - -Manual verification for generated Hermes Python: - -```bash -cargo test --test agent_test hermes -- --nocapture -cargo test --test hermes_lcm_bridge_test -- --nocapture -``` - -Expected: - -- Generated `plugin.yaml`, `schemas.py`, `schemas.json`, `tools.py`, `__init__.py`, and `skills/tokensave/SKILL.md` compile and register. -- `TokenSaveContextEngine` registers when Hermes exposes `register_context_engine`. -- The generated bridge calls `tokensave tool ... --json --args` and can parse nested MCP text JSON. -- Auxiliary summaries strip reasoning tags and fall back to deterministic truncation when all routes fail. - -## Self-Review - -- Spec coverage: The tasks cover in-place `sessions.db` migration, lossless raw storage, bounded derived search/display text, externalized payload containment, summary DAG lineage, lifecycle/frontier/debt state, deterministic Rust query APIs, additive MCP tools, compatibility for `tokensave_message_search`, Hermes generated Python lifecycle registration, subprocess bridge use, auxiliary LLM reasoning stripping/fallbacks, storage locality, ingest protection, diagnostics, and separation from codegraph/memory systems. -- Red-flag scan: The plan contains concrete file paths, test names, commands, expected red/green outcomes, commit checkpoints, and code/API sketches for each code-changing task. -- Type consistency: `LcmRawMessage`, `LcmStorageKind`, `LcmPayloadRef`, `LcmSourceRef`, `LcmSummaryNode`, `LcmLoadSessionRequest`, `LcmGrepRequest`, `LcmSummarizerMode`, and `TokenSaveContextEngine` are introduced before later tasks use them. -- Caveat: This plan chooses `/.tokensave/sessions.db` for non-local Hermes profile storage so the implementation has one concrete path rule. A later PyO3/native binding milestone remains outside this plan and should require measured bridge overhead before it starts. diff --git a/docs/superpowers/specs/2026-03-27-daemon-mode-design.md b/docs/superpowers/specs/2026-03-27-daemon-mode-design.md deleted file mode 100644 index 7b8b3d7a..00000000 --- a/docs/superpowers/specs/2026-03-27-daemon-mode-design.md +++ /dev/null @@ -1,197 +0,0 @@ -# Daemon Mode Design Spec - -> **Rebrand note:** The project has since been renamed **TraceDecay** (binary/crate `tracedecay`, MCP tools `tracedecay_*`). This dated design artifact keeps the TokenSave-era names it was written with. - -## Goal - -A background daemon that watches all tracked tokensave projects for file changes and automatically runs incremental syncs, keeping the code graph up-to-date without manual `tokensave sync` invocations. - -## CLI Surface - -``` -tokensave daemon # start (forks to background, writes PID file) -tokensave daemon --foreground # stay in foreground (for debugging / service managers) -tokensave daemon --stop # kill running daemon via PID file -tokensave daemon --status # check if running -tokensave daemon --enable-autostart # install launchd/systemd service -tokensave daemon --disable-autostart # remove service -``` - -All flags are mutually exclusive. `--foreground` is useful for debugging and when the daemon is managed by an external service manager. - -## Config - -Add `daemon_debounce` to `~/.tokensave/config.toml`: - -```toml -daemon_debounce = "15s" -``` - -A simple duration parser understands `s` (seconds) and `m` (minutes). Examples: `"15s"`, `"30s"`, `"1m"`, `"2m"`. Stored as `String` in `UserConfig`, parsed at runtime with `parse_duration()`. Default is `"15s"` if absent or unparseable. - -## Architecture - -### Components - -| Component | File | Responsibility | -|-----------|------|----------------| -| DaemonRunner | `src/daemon.rs` | Core event loop: project discovery, file watching, debounce, sync dispatch | -| PID management | `src/daemon.rs` | Write/read/check `~/.tokensave/daemon.pid` | -| Duration parser | `src/daemon.rs` | Parse `"15s"` / `"1m"` strings into `Duration` | -| Service installer | `src/daemon.rs` | Generate launchd plist (macOS) / systemd user unit (Linux) | -| CLI integration | `src/main.rs` | `Commands::Daemon` enum variant with flags | -| Doctor integration | `src/doctor.rs` | Check if daemon is running | - -### Data Flow - -1. **Startup:** Open global DB, read all project paths from `projects` table, set up a `notify::RecommendedWatcher` watching each project root recursively. - -2. **Project discovery (every 60s):** Re-read the global DB projects table. Add watchers for new projects. Remove watchers for projects no longer in the DB. - -3. **File change event:** When `notify` fires an event, determine which project it belongs to (by matching the event path against watched project roots). Mark that project as "dirty" and start/reset its per-project debounce timer. - -4. **Debounce fires (default 15s after last change):** Open `TokenSave::open()` for the dirty project, call `sync()`, log the result (files added/modified/removed, duration). Update global DB token count. - -5. **Filtering:** Ignore change events inside `.tokensave/`, `.git/`, `node_modules/`, `target/`, `.build/`, and other common build output directories. Also ignore events for files that don't match any supported language extension. - -### Self-Daemonizing - -On `tokensave daemon` (without `--foreground`): - -1. Fork the process using `fork()` (Unix) or equivalent -2. Detach from terminal (setsid, close stdin/stdout/stderr, redirect to log file `~/.tokensave/daemon.log`) -3. Write PID to `~/.tokensave/daemon.pid` -4. Enter the main event loop - -On `--foreground`: skip the fork, keep stderr/stdout attached, still write PID file. - -### PID File Management - -- **Path:** `~/.tokensave/daemon.pid` -- **Write:** On daemon start, write the PID as a plain integer -- **Read:** On `--stop` and `--status`, read the PID and check if the process is alive (`kill(pid, 0)` on Unix) -- **Stale detection:** If PID file exists but the process is dead, treat as not running. On start, overwrite stale PID files. -- **Cleanup:** On graceful shutdown (SIGTERM/SIGINT), delete the PID file - -### `--stop` - -Read PID file, check process alive, send SIGTERM. Wait up to 5 seconds for process to exit. If still alive after 5s, send SIGKILL. Remove PID file. - -### `--status` - -Read PID file, check process alive. Print: -- "tokensave daemon is running (PID: 12345)" or -- "tokensave daemon is not running" - -Exit code 0 if running, 1 if not. - -### `--enable-autostart` - -**macOS (launchd):** - -Write `~/Library/LaunchAgents/com.tokensave.daemon.plist`: -```xml - - - - - Label - com.tokensave.daemon - ProgramArguments - - /path/to/tokensave - daemon - --foreground - - RunAtLoad - - KeepAlive - - StandardOutPath - ~/.tokensave/daemon.log - StandardErrorPath - ~/.tokensave/daemon.log - - -``` - -Then run `launchctl load `. - -**Linux (systemd):** - -Write `~/.config/systemd/user/tokensave-daemon.service`: -```ini -[Unit] -Description=tokensave file watcher daemon - -[Service] -ExecStart=/path/to/tokensave daemon --foreground -Restart=on-failure -RestartSec=5 - -[Install] -WantedBy=default.target -``` - -Then run `systemctl --user enable --now tokensave-daemon.service`. - -### `--disable-autostart` - -**macOS:** `launchctl unload `, then delete the plist file. - -**Linux:** `systemctl --user disable --now tokensave-daemon.service`, then delete the unit file. - -## Error Handling - -- **Project path gone:** Skip it on next discovery cycle, don't crash. Log a warning once. -- **sync() failure:** Log the error, continue watching. Don't retry immediately — wait for next file change. -- **Global DB unreachable:** Retry on next 60s poll cycle. Keep existing watchers running. -- **Watcher limit exhaustion:** If the OS file watch limit is hit, log a warning and suggest increasing `fs.inotify.max_user_watches` (Linux) or similar. Continue watching already-registered projects. -- **Permission errors:** Log and skip the project. - -## Doctor Integration - -Add a "Daemon" section to `tokensave doctor` output: - -``` -Daemon - ✔ Daemon is running (PID: 12345) -``` -or -``` -Daemon - ! Daemon is not running — run `tokensave daemon` to start -``` - -Also check if autostart is enabled: -``` - ✔ Autostart enabled (launchd) -``` -or -``` - ! Autostart not configured — run `tokensave daemon --enable-autostart` -``` - -## Dependencies - -- `notify` crate (v7) — cross-platform file system watcher -- `nix` crate — Unix fork/setsid/signal handling (or use `libc` directly) - -## Logging - -The daemon writes to `~/.tokensave/daemon.log`. Log format: - -``` -[2026-03-27 14:32:01] started, watching 3 projects -[2026-03-27 14:32:16] synced /Users/foo/myproject — 2 added, 1 modified, 0 removed (45ms) -[2026-03-27 14:33:01] discovered new project: /Users/foo/another -[2026-03-27 14:33:05] synced /Users/foo/another — 0 added, 0 modified, 0 removed (12ms) -[2026-03-27 14:35:00] shutting down (SIGTERM) -``` - -## Out of Scope - -- Windows service support (can be added later) -- Remote/network file system watching -- Per-project debounce overrides (global config only) -- Web UI or dashboard diff --git a/docs/superpowers/specs/2026-06-09-tokensave-lcm-session-rewrite-design.md b/docs/superpowers/specs/2026-06-09-tokensave-lcm-session-rewrite-design.md deleted file mode 100644 index 093fbbbe..00000000 --- a/docs/superpowers/specs/2026-06-09-tokensave-lcm-session-rewrite-design.md +++ /dev/null @@ -1,241 +0,0 @@ -# TokenSave LCM Session Rewrite Design - -> **Rebrand note:** The project has since been renamed **TraceDecay** (binary/crate `tracedecay`, MCP tools `tracedecay_*`). This dated design artifact keeps the TokenSave-era names it was written with; read `tokensave` / `tokensave_*` as `tracedecay` / `tracedecay_*` when applying it to the current codebase. - -Date: 2026-06-09 -Branch: `feature/lcm-comparison` -Status: Approved design direction - -## Goals - -Port Hermes LCM into TokenSave by fully rewriting TokenSave's current simple session internals into an LCM implementation while preserving the useful public session-search surface. - -The design goals are: - -- Make the existing project-local `sessions.db` the authoritative LCM-capable session database through schema migrations, not by introducing a second parallel LCM database for TokenSave-managed sessions. -- Replace the simple provider-normalized session-message internals with lossless LCM-grade raw-message storage, summary-DAG, lifecycle/frontier, externalized-payload, and lineage state. -- Preserve the useful public behavior of provider-normalized transcript search, especially `tokensave_message_search`, while allowing its internals to be rebuilt on top of LCM-grade storage and indexed snippets. -- Remove authoritative session text caps for new writes. Caps may still exist for display snippets, FTS/index text, MCP response truncation, or safety-bounded rendering, but the authoritative stored raw session content must be lossless and expandable/recoverable. -- Use a hybrid ownership model: Rust owns durable/indexed state, migrations, deterministic query/DAG APIs, storage locality, and path-safety rules; the generated Hermes Python plugin owns Hermes `ContextEngine` lifecycle integration and Hermes auxiliary LLM calls. -- Start with the existing generated Python bridge that shells to `tokensave tool ... --json --args`, then consider PyO3 or native Python bindings later only if the bridge becomes a measured bottleneck. -- Keep TokenSave's codegraph context retrieval and fact memory systems separate from LCM session compression. - -## Non-Goals - -This design does not implement the port. It defines the target architecture and constraints for a later implementation. - -Non-goals are: - -- Do not add PyO3, maturin, or native Python bindings as the first milestone. -- Do not replace `src/context/builder.rs` with LCM. Codegraph context building remains graph/search/source retrieval for code intelligence. -- Do not repurpose `src/memory/*` fact storage into a summary DAG. The fact store remains structured, user/project memory; LCM stores transcript lineage and compression state. -- Do not require Hermes users to migrate to project-local storage for non-local installs. Storage locality follows current TokenSave and Hermes install rules. -- Do not keep the existing 256 KiB session-message cap as an authoritative storage limit for new session content. It can only survive as a derived-view limit for search/display/response safety. -- Do not preserve branch-only internal shapes that conflict with the approved design. The durable compatibility boundary is public session search and existing session DB data. - -## Architecture - -TokenSave should absorb Hermes LCM by turning the session layer into a Rust-owned LCM core with a Hermes-specific Python adapter. - -The high-level split: - -- Rust session/LCM core: - - Owns `sessions.db` schema, migrations, WAL/busy-timeout settings, storage path selection, path containment, and externalized-payload metadata. - - Replaces the simple provider-normalized message table as the authoritative raw-content layer with lossless LCM raw storage, while retaining compatibility projections/indexes for provider-normalized search. - - Exposes deterministic tool/API operations for search, load, describe, expand, expand-query support, compression status, lifecycle/frontier state, and DAG traversal. - - Preserves stable JSON outputs where existing MCP/Hermes tools depend on them. -- Generated Hermes Python plugin: - - Continues to be installed from `src/agents/hermes.rs`. - - Registers TokenSave tools, the `pre_llm_call` hook, and the `TokensaveMemoryProvider`. - - Hosts the Hermes `ContextEngine` adapter surface and calls TokenSave through `tokensave tool ... --json --args`. - - Calls Hermes `agent.auxiliary_client.call_llm` for LCM summarization and query expansion prompts that require an auxiliary model. - -This keeps durable correctness, migrations, and storage safety in Rust while avoiding a native extension boundary during the first port. - -## Storage/Migration Model - -The existing `sessions.db` becomes the LCM-capable session DB, and LCM replaces the simple session internals rather than sitting beside them as a second subsystem. - -Today, TokenSave stores project-local sessions at `crate::config::get_tokensave_dir(project_root).join("sessions.db")` through `src/sessions/cursor.rs`. `src/global_db.rs` creates provider-normalized tables: - -- `sessions(provider, session_id, project_key, project_path, title, started_at, ended_at, transcript_path, metadata_json, parent_session_id, is_subagent, agent_id, parent_tool_use_id)` -- `session_messages(provider, message_id, session_id, role, timestamp, ordinal, text, kind, model, tool_names, source_path, source_offset, metadata_json)` -- `session_messages_fts` over `text`, `role`, `kind`, `model`, and `tool_names` -- parse offsets and accounting tables used by existing transcript ingestion - -LCM migrations should rewrite/evolve this database instead of creating a separate TokenSave LCM DB. The migrated schema should support both existing transcript search behavior and LCM-grade raw storage: - -- Preserve `sessions` and provider-normalized message search as compatibility projections, not as the only authoritative session representation. -- Add or migrate to LCM raw-message identity with stable `store_id` ordering, conversation/session linkage, source/provider fields, role, lossless content reference, tool-call fields, timestamps, token estimates, pinned state, and metadata. -- Add summary DAG tables equivalent in capability to Hermes `summary_nodes`: `node_id`, session/conversation, depth, summary content/reference, token counts, source token counts, source ids, source type, source time window, expand hint, and FTS/search metadata. -- Add lifecycle/frontier state equivalent in capability to Hermes `lcm_lifecycle_state`: conversation id, current session, last finalized session, current frontier store id, last finalized frontier store id, debt markers, rollover/reset/maintenance timestamps, and update time. -- Add migration state/schema version tables in the existing DB layer so old `sessions.db` files can be upgraded idempotently. -- Add externalized-payload metadata that links placeholders in indexed text to payload files, content hashes, kind, role/tool metadata, session/conversation ownership, byte/char counts, and creation time. - -The current `MAX_SESSION_MESSAGE_TEXT_BYTES` cap of 256 KiB must be removed from authoritative storage for new writes. It may remain appropriate for FTS snippets, compatibility search rows, display previews, MCP response truncation, and safety-bounded rendering, but the LCM store must preserve full raw content either inline in a dedicated full-content column/table or by externalizing payloads and storing safe placeholders plus recoverable payload references. FTS should index bounded, safe text or snippets, not necessarily the full authoritative payload. - -Existing rows that were already capped cannot be made lossless retroactively. Migrations should carry those rows forward as best-effort legacy data, mark them as legacy/truncated when detectable, and ensure all new writes use lossless authoritative storage. - -Storage locality follows TokenSave install rules: - -- `--local` Hermes installs use project-local TokenSave storage under the project `.tokensave`, including the migrated `sessions.db` and local externalized payload directory. -- Non-local Hermes installs store LCM state under the Hermes profile/home location, matching Hermes LCM's current profile-local behavior. -- The path selector must be explicit so the same Rust APIs can open either the project-local DB or the Hermes-profile DB without guessing. - -## Rust Session/LCM Core - -The Rust core should become the source of truth for LCM state. It should not merely mirror the Python reference implementation table-for-table; it should provide the same semantics through TokenSave-native types and migrations. - -Core responsibilities: - -- Ingest provider-normalized transcript messages into the LCM raw-message model and derive provider-normalized search records from that model where useful. -- Guarantee lossless authoritative raw storage for new writes. Search snippets, FTS columns, MCP responses, and rendered previews may be capped, but `load`/`expand` APIs must be able to recover full content or the full externalized payload. -- Maintain append-first raw-message ordering and store-id lineage. Existing Hermes LCM allows one narrow rewrite for GC placeholders on externalized tool result rows; TokenSave should model any such mutation explicitly and keep source lineage intact. -- Keep the existing search path for `tokensave_message_search` over provider-normalized transcripts, while making LCM search able to combine raw messages and summary nodes with source/session filters. -- Implement summary-DAG persistence and deterministic traversal/expansion APIs in Rust. Python can request operations, but Rust should own source-id resolution, session ownership checks, FTS queries, pagination, and snippet construction. -- Track lifecycle/frontier state durably so session rollover, reset, maintenance debt, and current compaction frontier survive process restarts. -- Keep codegraph context retrieval independent. `ContextBuilder` continues to combine code search, graph traversal, and source extraction; LCM tools operate on conversation/session state. -- Keep fact memory independent. `MemoryCategory`, fact records, trust scores, entity links, and feedback are not summary nodes and should not be overloaded for transcript compression. - -The first Rust API surface can be tool-driven rather than a public library API. The important boundary is deterministic JSON in and out, because the generated Hermes plugin can call it through the existing bridge. - -## Hermes Python Context-Engine Adapter - -Hermes' current LCM reference owns a `ContextEngine`-style lifecycle: - -- `should_compress_preflight(messages)` is not a pure predicate. It ingests messages and can return `true` when the replay-safe view differs because ingest protection sanitized or externalized content. -- `compress(messages, current_tokens, focus_topic)` ingests, selects raw backlog outside the fresh tail, summarizes leaf chunks, optionally condenses summary nodes, advances the lifecycle frontier, assembles active prompt context, and returns sanitized replay messages. -- `on_session_start`, rollover/reset handling, auxiliary-session markers, and foreground session binding maintain lifecycle/frontier state. -- Tool calls can ingest live messages before search so current-turn content is discoverable. - -TokenSave's generated Hermes plugin should adapt this lifecycle to Rust-owned state rather than reimplementing storage in Python. The adapter should: - -- Register a Hermes context engine or equivalent hook surface for preflight, compression, session start, rollover, reset, and tool calls. -- Call TokenSave tools through the generated `tools.py` subprocess bridge with JSON args and JSON responses. -- Keep only process-local coordination in Python, such as active auxiliary session markers and bridge timeout/error handling, when that state does not need to be durable. -- Delegate all durable raw-message, DAG, frontier, payload, and search mutations to Rust. -- Preserve the existing TokenSave Hermes install/config behavior: generate plugin files, register `pre_llm_call`, register the memory provider, and refuse to overwrite an existing non-TokenSave Hermes memory provider. - -The adapter should also preserve Hermes' behavior that auxiliary LLM sessions are stateless for compression, so summarizer/query-expansion calls do not recursively ingest or compact themselves. - -## LLM Bridge - -Hermes LCM summarization depends on Hermes auxiliary model calls. The initial TokenSave port should keep those calls in generated Python: - -- Summary calls use `agent.auxiliary_client.call_llm` with task `compression`, messages containing the summary prompt, temperature, max tokens, optional timeout, and routed model settings. -- Reasoning tags such as ``, ``, ``, ``, and `` must be stripped before summary text is persisted. -- Model fallback and circuit breaker semantics should be preserved: try the configured primary/fallback route chain, stop hammering a failing route during cooldown, and fall back to deterministic truncation when LLM summarization cannot produce a smaller summary. -- Rust should store the resulting summary, source lineage, token accounting, and failure/status metadata. Python should not be the durable owner of summary nodes. - -For tools like `lcm_expand_query` that synthesize an answer from expanded context, Python may keep the auxiliary LLM call initially. Rust should provide deterministic context selection and expansion payloads so the LLM prompt is built from safe, session-authorized material. - -## Tool/API Surface - -`tokensave_message_search` remains stable or compatibly enhanced. - -Existing behavior to preserve: - -- Required `query`. -- Default provider of `cursor`. -- Optional `project_key`, `parent_session_id`, `include_subagents`, `scope`, and `limit`. -- Results containing provider-normalized session and message records with scores. -- The handler opens the project-local `sessions.db` and searches `session_messages_fts`. - -Enhancements should be additive at the public API boundary. Internally, search may be served from LCM-derived compatibility tables or indexed snippets rather than from the old capped text column. Result metadata may indicate legacy truncation, externalized payload references, or LCM availability, but existing callers should still be able to use the current fields. - -New LCM-oriented tools may be added rather than overloading `tokensave_message_search`: - -- `tokensave_lcm_status`: current session, lifecycle/frontier, compression status, debt, payload stats, and schema health. -- `tokensave_lcm_load_session`: ordered raw-message pages by explicit session id, with role/time filters, content slicing, and stable cursors. -- `tokensave_lcm_grep`: combined raw-message and summary-node search with current/session/all scope controls. -- `tokensave_lcm_describe`: session DAG overview or summary-node subtree metadata. -- `tokensave_lcm_expand`: expand a raw message, summary node, externalized payload, or source subtree with authorization checks and content pagination. -- `tokensave_lcm_expand_query`: deterministic context selection in Rust plus optional Hermes auxiliary synthesis in Python. -- `tokensave_lcm_compress` or internal equivalents: preflight/ingest/compress operations used by the Hermes adapter. - -Names can be finalized during implementation, but the behavioral split should remain: `tokensave_message_search` is compatibility transcript search; LCM tools expose compression-aware session state. - -## Ingest Protection/Security - -Hermes LCM contains important storage-boundary protections that should carry into TokenSave's Rust-owned store: - -- Externalize obvious large or binary-ish payloads, including data URI/base64 media, long base64 runs, oversized raw payloads, and large tool outputs. -- Store compact placeholders in indexed text while preserving recoverable payload content in full-content storage or externalized files with metadata and hashes. -- Keep externalized payload files private where possible, equivalent to `0700` directories and `0600` payload files. -- Reject payload refs that are not basenames. Do not allow `/`, `\`, parent traversal, symlink escape, or arbitrary absolute paths. -- Enforce storage root containment. Local installs stay under project `.tokensave`; non-local profile installs stay under the selected Hermes home/profile root. -- Keep session/conversation ownership checks on payload expansion so a ref from another session cannot be expanded casually. -- Preserve optional sensitive-pattern redaction as metadata-only/lossy when enabled, and make that lossiness visible in status/doctor output. -- Include integrity scans for missing/unreferenced payloads and SQLite/FTS health without previewing sensitive payload contents. - -The security model should treat search indexes, snippets, and rendered previews as derived, bounded, and safe-to-display. The authoritative raw content path must be lossless, separately controlled, paginated, and authorized. - -## Lifecycle/Compression Behavior - -The compression semantics should match Hermes LCM's observable behavior while moving durable state to Rust. - -Key behavior to preserve: - -- Preflight can ingest. It must be allowed to return compression-needed when ingest protection changes the replay view even if token thresholds alone would not require compression. -- Compression bypasses ignored sessions, stateless sessions, and auxiliary thread contexts. -- Compression respects a cooldown after boundary skips and can force overflow recovery when context pressure is critical. -- The fresh tail stays raw in active context. Raw backlog outside the fresh tail is summarized into leaf DAG nodes, with dynamic leaf chunk behavior where configured. -- Summary nodes preserve source ids, source token counts, time windows, depth, and expand hints so future `describe`/`expand` calls can recover lineage. -- Condensation can summarize lower-depth nodes into higher-depth nodes when needed, preserving a DAG rather than a flat rolling summary. -- Active prompt assembly combines the leading system anchor when present, relevant summary context, preserved objective context when needed, and the fresh tail. It must sanitize tool-call/tool-result pairing before returning replay messages. -- Rollover resets the active frontier for the new session while preserving the last finalized session/frontier for the conversation. -- Maintenance debt records when raw backlog could not yet be compacted, so deferred maintenance can run later instead of losing state. - -TokenSave should keep deterministic state transitions in Rust and leave nondeterministic summarization text generation to Hermes auxiliary LLM calls through Python. - -## Testing Strategy - -Testing should cover the design boundaries rather than only happy-path tool output. - -Required test areas: - -- Migration tests from existing `sessions.db` files with only `sessions`, `session_messages`, `session_messages_fts`, parse offsets, and parent/subagent columns into the LCM-capable schema. -- Migration tests proving already-capped legacy rows are carried forward as best-effort legacy data, marked as truncated/legacy where detectable, and never treated as newly lossless content. -- Compatibility tests proving `tokensave_message_search` retains current provider/project/subagent filtering and result shape. -- Raw-storage tests proving new content above 256 KiB is preserved authoritatively and recoverable through load/expand APIs while search indexes, snippets, MCP responses, and display renderers use safe bounded views. -- Regression tests proving no new authoritative session write path uses `MAX_SESSION_MESSAGE_TEXT_BYTES` or any replacement cap before durable raw storage/externalization. -- Externalization tests for large tool output, data URI/base64 payloads, basename-only refs, private permissions, root containment, missing payloads, and cross-session expansion denial. -- Lifecycle tests for session start, rollover, reset, frontier advance, last-finalized frontier preservation, maintenance debt, and restart recovery. -- Compression tests with fake Hermes auxiliary LLM calls covering leaf compaction, condensation, fallback chain, circuit breaker, deterministic truncation fallback, reasoning stripping, and active-context assembly. -- Generated Hermes plugin tests proving install layout, config editing, memory provider registration, `pre_llm_call`, JSON bridge calls, and non-overwrite behavior for existing memory providers. -- Tool tests for load/grep/describe/expand/expand-query/status using deterministic fixtures. -- Regression tests that `src/context/builder.rs` and `src/memory/*` remain independent from LCM session compression. - -## Rollout/Backward Compatibility - -Rollout should be incremental but converge on one session store design. - -Compatibility commitments: - -- Existing `sessions.db` files are migrated in place with idempotent schema migrations. -- Existing capped rows are preserved as best-effort legacy data. New session writes after migration must be lossless in the LCM raw store. -- Existing provider-normalized transcript ingestion continues to support `sessions` and `session_messages` behavior, but those simple internals become compatibility projections over the LCM-grade session store where practical. -- `tokensave_message_search` remains stable or only compatibly enhanced. -- Existing local installs continue to store under project `.tokensave`; non-local Hermes installs continue to store under the Hermes profile/home. -- Existing Hermes plugin generation remains the installation mechanism, but its generated Python grows the LCM adapter and tool bridge calls. - -Rollout shape: - -- Introduce schema migrations and read-only LCM inspection APIs first, including proof that new authoritative writes are lossless and old capped rows are flagged as legacy where detectable. -- Add Hermes adapter preflight/ingest with compression disabled or status-only until storage and search compatibility are verified. -- Enable compression in Hermes after raw storage, externalization, lifecycle, and active-context assembly tests pass. -- Measure bridge overhead before considering PyO3/native bindings. A later native milestone should be justified by latency, packaging, or reliability evidence. - -## Open Questions - -- Should non-local Hermes-profile LCM storage reuse a file named `sessions.db` for consistency, or keep a Hermes-specific filename while using the same migrated schema? -- What is the exact Rust table layout for full raw content: inline full-content table, externalized-by-default payload table, or a hybrid threshold policy? -- Which LCM operations should be public MCP tools versus internal Hermes-plugin-only tools? -- Should TokenSave expose LCM summary nodes to non-Hermes providers once their transcripts are indexed, or initially limit compression lifecycle to Hermes sessions? -- How much of `lcm_expand_query` synthesis should remain Python-only after context selection, and what JSON contract should Rust provide for reproducible prompt construction? - -## Self-Review - -This spec intentionally describes one plan: fully rewrite the existing TokenSave session internals into a lossless LCM-capable store inside the existing `sessions.db`, with Rust owning durable state and generated Python owning Hermes lifecycle/auxiliary LLM integration. It rejects the earlier separate-DB starting point, rejects capped authoritative storage for new writes, and rejects PyO3 as the first milestone. - -No placeholders or TBD markers remain. The open questions are bounded design choices for implementation, not unresolved contradictions in the approved architecture.