Add checkpoint-backed AI transcript reconciliation planner

[jmagar]

Summary

Add a checkpoint-backed AI transcript reconciliation planner so cortex ai watch can recover from lossy watcher overflow without repeatedly doing expensive unchanged-file work across the whole transcript corpus.

This is a follow-up to the immediate scan-storm mitigation in bead syslog-mcp-jumyo, which bounded overflow rescans to a recent mtime window and rate-limited retries. That fix stops the observed incident class, but recovery is still scan-based: index_roots_with_options walks roots, collects supported files, sorts/dedups them, then applies since_mtime_nanos per file.

The MVP should build on existing scanner checkpoints, not introduce a Merkle-first parallel index. transcript_sources remains the canonical per-file identity/checkpoint owner; new root/generation state describes reconciliation.

Tracking epic: syslog-mcp-ssxk4

Motivation

The suspicious workload during the dookie incident was repeated AI transcript indexing over roughly 5,553 transcript files, with one index_ai_roots run around 152801ms and DB lock fallout. The current bounded fallback is safe, but future recovery can be cheaper and more observable after warm state.

The realistic goal is not “no filesystem work after overflow.” Watcher overflow is lossy, and without another reliable journal Cortex may still need to enumerate/stat files. The goal is:

Warm overflow reconciliation may enumerate/stat broadly, but unchanged transcript files must not be opened, hashed, parsed, or rewritten in the main DB.

Expensive work and DB writes should be proportional to changed/new/missing/stale/error files, not total known files.

Existing Evidence

• src/ai_watch.rs: overflow recovery uses a 5-minute lookback, 60-second retry interval, and calls index_ai_roots(... since ...).
• src/app/services/ai_indexing.rs: service converts since to since_mtime_nanos and runs scanner work via DB execution.
• src/scanner.rs: root indexing currently collects supported files, sorts/dedups, optionally filters by mtime, and indexes file-by-file.
• src/scanner/checkpoint.rs and DB schema: transcript_sources already stores canonical_path, source_kind, file_size, file_mtime, content_hash, last_offset, last_indexed_at, and last_error.
• cortex ai doctor currently has paths that can iterate checkpoint paths and call Path::exists() for missing-checkpoint state.

MVP Scope

Implement a checkpoint-backed reconciliation model with:

• A root/generation model linked to transcript_sources.
• A planner API that classifies files for reconciliation.
• Watch overflow wiring that uses the planner when available and falls back to the current bounded mtime scan on failure.
• DB-backed manifest/checkpoint health diagnostics.
• Counter-based tests plus an incident-scale manual/ignored benchmark.

Out of MVP unless benchmarks later prove otherwise:

• Full content Merkle tree.
• First-class move/rename detection; treat moves as missing + new.
• Raw transcript diagnostic output.
• Replacing the existing bounded/rate-limited overflow fallback.
• Trusting manifest rows as authorization to access arbitrary paths.

Planner Contract

Layer ownership:

• Scanner owns path discovery, current path-safety validation, and reconciliation planning.
• Service owns async DB execution and API/CLI shaping.
• Watcher chooses “planner overflow reconciliation” with bounded mtime fallback; it must not inspect manifest tables directly.

Planner output classes:

• changed
• new
• missing / tombstoned
• stale_or_error
• fallback

Treat move/rename as missing + new.

Counters to expose for tests/logs/diagnostics:

• total_files
• stat_checked
• scheduled
• opened
• hash_verified
• parsed_records
• db_writes
• write_lock_ms
• elapsed_ms
• skipped_by_manifest
• fallback_reason

State Transitions To Define

Before implementation, define how reconciliation state changes for:

• Unchanged metadata/hash.
• Append success.
• Rewrite success, including preserved-mtime rewrite detection.
• Parse error.
• Storage-blocked chunk.
• Force reset/reimport.
• Missing/tombstone.
• Manifest absent/stale/corrupt/partial migration/planner read error fallback.

Storage-blocked or partial-failure paths must not falsely mark files fresh.

Schema Requirements

Prefer transcript_roots plus root/generation fields linked to transcript_sources, for example:

• root_id
• root-relative path/identity where useful
• last_seen_generation
• manifest_state
• missing_since
• optional platform fingerprint fields such as ctime/inode/dev for preserved-mtime rewrite detection

Avoid duplicating existing file_size/file_mtime metadata unless the design explicitly introduces an observed-but-not-indexed state.

Add explicit indexes and query-plan checks for:

• Root + canonical or relative path lookup.
• Root + generation lookup.
• Root + manifest state lookup.
• Stale/missing queries and anti-joins.

Avoid SQL that looks clean but full-scans under load.

Security And Privacy Acceptance Criteria

• Manifest rows are advisory only. Every scheduled path must be revalidated against current canonical root, symlink, file-type, file-size, and supported-format policy before stat/open/hash/parse.
• Manifest metadata needed for correctness may be stored, but read diagnostics expose aggregate counts and bounded reason codes by default.
• Full paths, per-file manifest states, tombstone path lists, inode/dev/ctime, parse previews, record keys, and transcript-content-derived identifiers require admin/verbose or local-only mode if exposed at all.
• Manifest failure/corruption/partial migration falls back only to the current bounded mtime scan, never an unbounded scan.
• Overflow logs should avoid sampled path lists by default because logs become persistent syslog records.

Performance Acceptance Criteria

• Warm no-change reconciliation over an incident-scale corpus must not open/hash/parse thousands of unchanged files.
• Warm no-change reconciliation must perform zero or near-zero main-DB writes; any necessary generation bookkeeping should be batched in one short transaction.
• No filesystem walking, opening, hashing, or parsing while holding SQLite write lock.
• Planner query paths have indexes and tests that guard against accidental full scans.
• Watcher overflow logs/counters show notify/internal/pending overflow reason where possible, planner elapsed time, scheduled/opened/hashed/parsed counts, and fallback reason.

Validation Plan

Normal CI / cargo test:

• Migration/idempotency tests for root/generation schema.
• Query-plan tests for manifest lookup paths, without depending on brittle exact SQLite EQP text.
• Counter-based scanner/planner tests for unchanged, append, rewrite, parse error, storage-blocked, force reset, missing/tombstone, symlink replacement, stale malicious manifest path, broad-root rejection, and bounded fallback.
• Watch overflow tests proving rate limit remains and planner/fallback counters are emitted.
• Diagnostic tests proving read-safe defaults and partial-failure behavior.

Manual or ignored incident-scale benchmark:

• Cold initial scan.
• Warm no-change reconciliation over ~5,553 synthetic transcript files.
• Warm small-change reconciliation.

Benchmark counters should include total_files, stat_checked, scheduled, opened, hash_verified, parsed_records, db_writes, write_lock_ms, elapsed_ms, and fallback_reason.

Bead Breakdown

• syslog-mcp-ssxk4.1 — Design AI transcript reconciliation schema and invariants.
• syslog-mcp-ssxk4.2 — Persist manifest updates during AI transcript indexing.
• syslog-mcp-ssxk4.3 — Implement manifest-backed transcript reconciliation planner.
• syslog-mcp-ssxk4.4 — Wire AI watch overflow recovery to manifest reconciliation.
• syslog-mcp-ssxk4.5 — Expose manifest health in AI diagnostics.
• syslog-mcp-ssxk4.6 — Benchmark and document AI transcript manifest reconciliation.

Validated bead graph: .1 -> .2 -> .3 -> (.4, .5) -> .6.

Lavra Engineering Review Applied

This section applies the engineering-review feedback directly to the GitHub issue so the issue can be implemented without reading the beads separately.

Architecture Findings Applied

1. Keep transcript_sources canonical. The implementation must not create a parallel file index whose rows can disagree with transcript_sources. If a separate manifest table is added, it must be linked by strict FK/uniqueness to transcript_sources(id).
2. Model reconciliation as root/generation state plus per-source manifest state. Do not update every unchanged source row on every warm reconciliation just to mark last_seen_generation; that would convert the fix into a DB-write storm.
3. Generation bookkeeping should be coarse and batched: use a reconciliation run/root generation record plus changed/new/missing/stale source updates, not per-file writes for no-change files.
4. Explicit single-file scans need their own path through the planner. They may not require a root row, but they must still preserve checkpoint/import semantics and safety validation.
5. Directory rollups/Merkle nodes, if introduced later, are pruning hints only. Per-file metadata/hash and path validation remain the correctness boundary.

Simplicity Findings Applied

1. MVP should not include full Merkle trees, move/rename detection, raw diagnostic dumps, or a second diagnostics subsystem.
2. Planner output classes stay intentionally small: changed, new, missing_or_tombstoned, stale_or_error, and fallback.
3. Use one checkpoint-store/reconciliation helper for manifest writes. Avoid scattering manifest SQL through scanner, service, watcher, CLI, and API layers.
4. Keep lower-level list/checkpoint/doctor commands for diagnostics, but normal watcher recovery should call one scanner-owned planner API.

Security Findings Applied

1. Manifest rows are advisory. Every scheduled path must be revalidated against current canonical accepted roots, symlink policy, file type, file size, and supported-format rules before stat/open/hash/parse.
2. Manifest corruption, stale schema, partial migration, or planner read errors may only fall back to the existing bounded mtime scan. They must never trigger unbounded whole-root recovery.
3. Read diagnostics should expose aggregate counts and bounded reason codes by default. Full paths, tombstone path lists, inode/dev/ctime, parse previews, record keys, and transcript-content-derived hashes require admin/verbose or local-only mode.
4. Overflow logs must not include sampled path lists by default because Cortex logs are persistent records.

Performance Findings Applied

1. Warm no-change reconciliation may enumerate/stat broadly, but it must not open, hash, parse, or rewrite thousands of unchanged transcript files.
2. No filesystem walking, opening, hashing, or parsing while holding the SQLite write lock.
3. Warm no-change reconciliation should perform zero or near-zero main-DB writes; any required root/generation bookkeeping must be a short batched transaction.
4. Add query-plan tests for root/path lookup, generation lookup, manifest-state lookup, stale/missing anti-joins, and fallback candidate selection.
5. Benchmarks must distinguish cold initial scan, warm no-change reconciliation, and warm small-change reconciliation. The warm no-change run should be far below the 152801ms incident run and should prove scheduled/opened/hash_verified/parsed_records remain near zero for unchanged files.

Failure Modes To Test

Codepath	Failure Mode	Required Behavior	Test Required
Schema migration	Partial migration leaves manifest tables missing or inconsistent	Planner reports fallback and watcher uses bounded mtime scan only	Yes
Root generation bookkeeping	Warm no-change run updates every known file row	Reject via counters/benchmark; only batch coarse generation state	Yes
Planner lookup	Query plan full-scans transcript_sources under large corpus	Indexed lookup or test failure	Yes
Scheduled path	Manifest row points outside accepted transcript root	Revalidate and reject; do not open path	Yes
Symlink replacement	File becomes symlink after manifest observation	Revalidate and skip/error safely	Yes
Preserved-mtime rewrite	Size+mtime appear unchanged but content changed	Use stored hash/fingerprint fallback before skipping; never silently miss changed content	Yes
Storage blocked	Storage guard blocks writes mid-file	Do not mark source fresh or advance completed generation	Yes
Parse error	Bad row keeps retrying in a hot loop	Persist bounded error state and expose repair signal	Yes
Watch overflow	Notify overflow loses path list	Planner reconciliation runs; on planner failure use bounded mtime fallback	Yes
Hooked diagnostics	ai doctor manifest read fails	Preserve OS/process status and include bounded manifest failure reason	Yes

NOT In MVP

• Full content Merkle tree.
• First-class move/rename detection; moves are missing plus new.
• Raw transcript diagnostic output.
• Replacing the current bounded/rate-limited overflow fallback.
• Unbounded recovery scans.
• Diagnostic surfaces that expose full paths/tombstones/fingerprints to read-only callers by default.

Updated Implementation Checklist

• Add root/generation schema linked to transcript_sources without duplicating canonical file identity.
• Define manifest state transitions for unchanged metadata/hash, append success, rewrite success, parse error, storage-blocked chunk, force reset, missing/tombstone, manifest absent/stale/corrupt, and fallback.
• Implement scanner-owned planner API returning classified work items and counters.
• Wire watcher overflow to planner first and bounded mtime fallback second.
• Add DB-backed manifest health to diagnostics without broad filesystem checks on the fast path.
• Add fast unit/query-plan/counter tests and ignored/manual incident-scale benchmark.
• Update docs with cold scan vs warm reconciliation vs fallback behavior.

Self-Contained Referenced Bead Context

The GitHub issue is authoritative for implementation. The bead references below are included in full so implementers do not need to open Beads for context.

Referenced Bead: syslog-mcp-ssxk4

○ syslog-mcp-ssxk4 [EPIC] · Add checkpoint-backed AI transcript reconciliation planner   [● P2 · OPEN]
Owner: Jacob Magar · Type: epic
Created: 2026-06-23 · Updated: 2026-06-23
External: gh-93

DESCRIPTION
## Problem
The immediate cortex-ai-watch scan storm was fixed in `syslog-mcp-jumyo` by bounding overflow rescans to a recent mtime window and rate-limiting retries. That stops the observed dookie failure mode, but overflow recovery is still fundamentally scan-based: when watcher events are missed or coalesced, Cortex must walk transcript roots and decide from filesystem metadata which files deserve indexing.

A persisted transcript manifest, optionally with Merkle-style per-directory rollups, should make future reconciliation cheaper and more robust by giving Cortex a durable view of known roots, known files, last-seen generations, metadata fingerprints, and missing/tombstoned paths.

## Objective
Design and implement a manifest-backed AI transcript reconciliation path that reuses the existing checkpoint/import machinery, avoids repeated full-ish scans under watcher overflow, and remains safe when files are rewritten, appended, deleted, or moved.

## Non-goals
- Do not replace the current bounded/rate-limited overflow fix.
- Do not make every watcher event compute full file content hashes.
- Do not weaken transcript root/path safety rules.
- Do not add another independently maintained indexing system beside `transcript_sources` without a migration/ownership story.
- Do not expose raw transcript contents in diagnostics.

## Sources
- Prior immediate-fix bead: `syslog-mcp-jumyo` (`fix: bound AI watch overflow rescans`).
- `src/ai_watch.rs`: overflow recovery calls `index_ai_roots(path, false, since)` via `run_rescan`.
- `src/app/services/ai_indexing.rs`: service API converts `since` into `since_mtime_nanos` and runs scanner work through `run_db`.
- `src/scanner.rs`: `index_roots_with_options` currently collects supported files by walking roots, sorts/dedups, then applies optional mtime filtering before indexing each file.
- `src/scanner/checkpoint.rs`: `transcript_sources` already stores per-file metadata (`file_size`, `file_mtime`, `content_hash`, `last_offset`, `last_error`) and powers checkpoint/doctor diagnostics.
- `docs/CLI.md` and `README.md`: current AI index/watch/checkpoint behavior and operator contracts.

## Acceptance Criteria
- Overflow reconciliation can use a persisted manifest/planner instead of blindly rewalking and testing every known transcript file on every recovery pass.
- Existing append-only optimization, duplicate suppression, parse-error persistence, storage guardrails, and `--force` semantics continue to work.
- Manifest tables/columns have indexed lookup paths for root, canonical path, generation, and stale/missing detection.
- The design explicitly chooses between simple manifest, directory rollup hashes, or a Merkle-style tree, with documented tradeoffs.
- `cortex ai doctor`/checkpoint diagnostics expose enough manifest health to debug stale roots without scanning all files during health checks.
- Tests cover append, rewrite with preserved mtime, delete/missing, overflow-style reconciliation, migration/idempotency, and broad-root/symlink safety.
- Documentation explains the manifest model, recovery behavior, and fallback behavior when the manifest is absent or stale.

## Open Design Questions
- Should the manifest extend `transcript_sources`, add root/generation tables, or introduce a new normalized manifest table linked to sources?
- Is a full Merkle tree worth the complexity, or are per-directory aggregate fingerprints enough for the observed 5.5k-file scale?
- What is the right fallback when manifest state is missing, corrupted, or older than the scanner schema?
- How should manifest pruning interact with `cortex ai prune-checkpoints --missing`?


LABELS: ai-indexing, performance, reliability

CHILDREN
  ↳ ○ syslog-mcp-ssxk4.1: Design AI transcript reconciliation schema and invariants ● P2
  ↳ ○ syslog-mcp-ssxk4.2: Persist manifest updates during AI transcript indexing ● P2
  ↳ ○ syslog-mcp-ssxk4.3: Implement manifest-backed transcript reconciliation planner ● P2
  ↳ ○ syslog-mcp-ssxk4.4: Wire AI watch overflow recovery to manifest reconciliation ● P2
  ↳ ○ syslog-mcp-ssxk4.5: Expose manifest health in AI diagnostics ● P3
  ↳ ○ syslog-mcp-ssxk4.6: Benchmark and document AI transcript manifest reconciliation ● P3
  ◐ 0/6 complete (0%)

COMMENTS
  2026-06-23 19:28 Jacob Magar
    RESEARCH SUMMARY:
    - FACT: The immediate overflow fix remains scan-based by design: `cortex ai watch` uses a 5-minute overflow lookback, 60-second retry interval, and calls `index_ai_roots(... since ...)`. This is a safe mitigation, not a durable reconciliation manifest.
    - FACT: `index_roots_with_options` still walks roots, collects all supported files, sorts/dedups, then applies `since_mtime_nanos` per file. The persisted manifest work should reduce expensive unchanged-file work after that warm state, not pretend lossy watcher overflow can be resolved with no reconciliation.
    - FACT: `transcript_sources` is already the canonical per-file checkpoint/identity table (`canonical_path`, `source_kind`, `file_size`, `file_mtime`, `content_hash`, `last_offset`, `last_indexed_at`, `last_error`). The manifest must extend or link to it instead of creating a second file truth.
    - PATTERN: A full Git-style content Merkle tree is likely overkill for the P2 follow-up at ~5,500 mutable append-heavy transcript files. Prefer a simple persisted root/generation/file manifest first; directory rollups may be pruning aids, not the correctness boundary.
    - PATTERN: Warm reconciliation may remain O(total files) for enumerate/stat, because directory mtimes do not detect file appends. Acceptance should target expensive work and DB writes being O(changed + new + missing + stale/error), with explicit counters for discovered/stat_checked/scheduled/hash_verified/skipped_by_manifest/fallback_reason.
    - PATTERN: Every manifest planner lookup needs explicit indexed paths and query-plan checks for root, canonical/relative path, generation, state, and stale/missing detection. Prior Cortex learning (`syslog-mcp-2rap`) showed a JOIN rewrite can still full-scan without the right filter-column index.
    - FACT: Current `ai doctor` missing-checkpoint detection iterates all `transcript_sources.canonical_path` values and calls `Path::exists()`. Manifest diagnostics should make common health checks DB-backed and reserve broad filesystem verification for explicit prune/repair paths.
    - FACT: Current service-layer `ai_watch_status` already preserves partial OS/process data when DB health fails. Manifest diagnostics should be additive and must not regress this behavior.
  2026-06-23 19:33 Jacob Magar
    DECISION: Applied engineering review feedback from architecture, simplicity, security, and performance passes.
    
    LEARNED/RISK SUMMARY:
    - Reframe the MVP as a checkpoint-backed reconciliation planner, not a Merkle-first manifest. `transcript_sources` remains the canonical per-file identity owner; new root/generation state describes reconciliation, not a second file truth.
    - Full content Merkle tree, first-class move detection, raw transcript diagnostic output, and replacing the bounded overflow fallback are out of MVP unless later benchmark evidence justifies them.
    - Warm overflow reconciliation may enumerate/stat all files, but unchanged files must not be opened, hashed, parsed, or rewritten in the main DB.
    - Planner contract must define inputs, output classes, counters, fallback reasons, and layer ownership: scanner owns path discovery/safety/planning, service owns async DB execution, watcher chooses planner overflow recovery with bounded mtime fallback.
    - Manifest rows are advisory only. Every scheduled path must be revalidated against current root, symlink, file-type, file-size, and supported-format policy before stat/open/hash/parse.
    - Directory rollups or Merkle nodes, if added later, are pruning hints only; per-file metadata/hash remains the correctness boundary before skipping indexing.
    - No filesystem walking, hashing, or parsing while holding the SQLite write lock; warm no-change reconciliation must not perform per-file main-DB updates.
    - Diagnostics must be additive, DB-backed by default, partial-failure tolerant, and read-safe by default.
  2026-06-23 19:35 Jacob Magar
    GITHUB ISSUE: https://github.com/jmagar/cortex/issues/93

Referenced Child Beads

syslog-mcp-ssxk4.1

○ syslog-mcp-ssxk4.1 · Design AI transcript reconciliation schema and invariants   [● P2 · OPEN]
Owner: Jacob Magar · Type: task
Created: 2026-06-23 · Updated: 2026-06-23

DESCRIPTION
## What
Design the persisted manifest data model for AI transcript roots and files, including whether to extend `transcript_sources` or add linked manifest/root/generation tables.

## Context
`transcript_sources` already tracks per-file `canonical_path`, `source_kind`, `file_size`, `file_mtime`, `content_hash`, `last_offset`, and error state. The missing piece is durable root/reconciliation state: root identity, scan generation, last-seen marker, stale/missing detection, and optionally directory aggregate hashes.

## Decisions
- Prefer reusing `transcript_sources` for canonical per-file identity unless the design proves a separate file table is safer.
- Add root/generation metadata rather than a parallel cache with no ownership relationship.
- Treat Merkle-style hashes as optional optimization; document whether simple per-directory aggregate fingerprints are sufficient.
- Require indexed lookup paths for root, canonical path, generation, stale/missing status, and last observed metadata.
- Preserve current path trust boundary: no broad-root scanning, no symlink traversal, no raw transcript content in diagnostics.

## Testing
- Migration creates tables/indexes idempotently on a fresh DB and upgraded DB.
- Schema tests prove uniqueness and foreign-key behavior for root/path/source relationships.
- Tests prove manifest metadata can represent append, rewrite, missing file, parse error, and force reimport states.

## Validation
- `cargo test scanner::checkpoint` or equivalent targeted migration/checkpoint tests pass.
- `cortex ai doctor --json` remains compatible or has documented additive fields only.
- Query plans for stale/missing/root lookup use indexes rather than full table scans.

## Files
- `src/db/*` migration/schema code
- `src/scanner/checkpoint.rs`
- `src/scanner/checkpoint_tests.rs`
- `src/scanner.rs`
- `docs/CLI.md`
- `README.md`

## Dependencies
None. This is the design root for the implementation plan.

## References
- Epic: `syslog-mcp-ssxk4`
- Existing checkpoint model: `src/scanner/checkpoint.rs`
- Existing scanner metadata: `src/scanner.rs`
- Prior immediate fix: `syslog-mcp-jumyo`


LABELS: ai-indexing, db, design, performance, reliability

PARENT
  ↑ ○ syslog-mcp-ssxk4: (EPIC) Add checkpoint-backed AI transcript reconciliation planner ● P2

BLOCKS
  ← ○ syslog-mcp-ssxk4.2: Persist manifest updates during AI transcript indexing ● P2

COMMENTS
  2026-06-23 19:28 Jacob Magar
    RESEARCH FINDINGS TO APPLY:
    - FACT: Existing `transcript_sources` already owns canonical per-file identity and metadata. Schema design should add `transcript_roots` plus root/generation fields on `transcript_sources`, or a thin linked manifest table; avoid a parallel file index.
    - PATTERN: Include root-relative identity and generation state: `root_id`, `relative_path` where useful, `last_seen_generation`, `manifest_state`, `missing_since`, `last_observed_size`, `last_observed_mtime_ns`, and Unix `ctime_ns`/inode/dev where available.
    - PATTERN: Preserved-mtime rewrites need a detection story: use ctime/file-id fingerprint where available, with hash fallback for unsupported platforms or suspicious equal fingerprints.
    - DECISION FEEDBACK: Add explicit P2 non-goal: no full content Merkle tree unless benchmarks prove simple metadata planning insufficient. Directory rollups are optional pruning aids only.
    - VALIDATION FEEDBACK: Add EXPLAIN/query-plan checks for `(root_id, canonical_path or relative_path)`, `(root_id, last_seen_generation)`, `(root_id, manifest_state)`, and stale/missing queries.
  2026-06-23 19:33 Jacob Magar
    ENGINEERING REVIEW FEEDBACK:
    - RISK: Avoid duplicating existing metadata. Prefer `transcript_roots` plus `root_id`, `relative_path`, `last_seen_generation`, `manifest_state`, `missing_since`, and optional platform fingerprint fields linked to `transcript_sources`. Do not duplicate `file_size`/`file_mtime` as last-observed fields unless the design defines a separate observed-but-not-indexed state.
    - RISK: `transcript_sources` remains canonical file identity. Any separate manifest file table must have a strict FK/uniqueness contract back to `transcript_sources(id)`.
    - RISK: Manifest rows are not authorization. Planner paths must be revalidated against canonical accepted roots and symlink/file policy before use.
    - RISK: Store ctime/inode/dev only for correctness; do not expose them in read diagnostics by default.
    - VALIDATION: Add explicit indexes and query-plan checks for root/path, generation, state, stale/missing lookups.

syslog-mcp-ssxk4.2

○ syslog-mcp-ssxk4.2 · Persist manifest updates during AI transcript indexing   [● P2 · OPEN]
Owner: Jacob Magar · Type: task
Created: 2026-06-23 · Updated: 2026-06-23

DESCRIPTION
## What
Update the scanner/checkpoint write path so every successful, skipped, failed, forced, and parse-error indexing outcome updates the persisted manifest consistently.

## Context
`index_file_with_options` already computes current/final `FileMetadata`, uses append-offset optimization when possible, stores content hashes after successful stable reads, and records parse errors. The manifest must follow those same state transitions instead of inventing a second truth source.

## Decisions
- Manifest writes should happen in the same transactional boundaries as source/import/checkpoint updates where practical.
- Skipped unchanged files must update last-seen/generation state without rewriting content hashes unnecessarily.
- Parse errors should preserve enough manifest state to avoid repeated hot-loop retries while still surfacing repair work.
- `--force` must reset/import manifest state consistently with existing checkpoint reset behavior.
- Avoid expensive hashing in event/watcher hot paths unless size/mtime/offset evidence says it is needed.

## Testing
- Existing idempotency, append-only, rewrite, preserved-mtime, parse-error, and force-reimport tests continue to pass.
- New tests verify manifest state after unchanged skip, append success, rewrite fallback, parse failure, storage-blocked chunk, and force reset.
- Tests verify manifest writes do not leave stale generation markers after partial failures.

## Validation
- Targeted scanner/checkpoint tests pass.
- `cargo test` passes before implementation is considered done.
- SQLite write contention does not increase materially in synthetic multi-file indexing tests.

## Files
- `src/scanner.rs`
- `src/scanner/checkpoint.rs`
- `src/scanner/checkpoint_tests.rs`
- `src/scanner_tests.rs`
- `src/db/*` migration/schema files

## Dependencies
Depends on `syslog-mcp-ssxk4.1` for finalized schema/invariants.

## References
- `src/scanner.rs` append/rewrite logic around `FileMetadata`, `flush_chunk`, and checkpoint updates.
- `src/scanner/checkpoint.rs` source metadata and reset/list/doctor behavior.


LABELS: ai-indexing, db, performance, reliability, scanner

PARENT
  ↑ ○ syslog-mcp-ssxk4: (EPIC) Add checkpoint-backed AI transcript reconciliation planner ● P2

DEPENDS ON
  → ○ syslog-mcp-ssxk4.1: Design AI transcript reconciliation schema and invariants ● P2

BLOCKS
  ← ○ syslog-mcp-ssxk4.3: Implement manifest-backed transcript reconciliation planner ● P2

COMMENTS
  2026-06-23 19:28 Jacob Magar
    RESEARCH FINDINGS TO APPLY:
    - PATTERN: Manifest writes should share the scanner/checkpoint/import transaction boundary where practical; current checkpoint writes happen in the import flow, so do not add an eventually-consistent side channel unless explicitly justified.
    - PATTERN: Skipped unchanged files should update generation/last-seen state without recomputing content hashes or rewriting unchanged import rows.
    - FACT: Append optimization and rewrite detection currently depend on per-file metadata, stored hash, and `last_offset`; manifest updates must preserve those semantics.
    - VALIDATION FEEDBACK: Add tests for partial failure and storage-blocked cases to ensure generation markers do not falsely mark unprocessed files as fresh.
  2026-06-23 19:33 Jacob Magar
    ENGINEERING REVIEW FEEDBACK:
    - RISK: Define state transitions for unchanged metadata/hash, append success, rewrite success, parse error, storage-blocked chunk, force reset, missing/tombstone, and fallback. Each outcome must specify whether it advances generation, changes manifest_state, records observation metadata, or leaves prior state intact.
    - RISK: Keep one checkpoint-store helper for generation/manifest state instead of scattered writes.
    - PERFORMANCE GATE: Do not update every unchanged file just to mark last_seen_generation. Warm no-change reconciliation should perform zero or near-zero main-DB writes; any necessary generation bookkeeping must be batched in one short transaction.
    - LOCK GATE: No filesystem walking, hashing, opening, or parsing while holding SQLite write lock.
    - FAILURE GATE: Storage-blocked or partial-failure paths must not mark files fresh.

syslog-mcp-ssxk4.3

○ syslog-mcp-ssxk4.3 · Implement manifest-backed transcript reconciliation planner   [● P2 · OPEN]
Owner: Jacob Magar · Type: task
Created: 2026-06-23 · Updated: 2026-06-23

DESCRIPTION
## What
Build a reconciliation planner that compares current transcript roots against the persisted manifest and returns only paths that need indexing, missing/tombstoned handling, or fallback verification.

## Context
`index_roots_with_options` currently walks roots, collects all supported files, sorts/dedups, and optionally skips older mtimes. That helped the incident fix but still means overflow recovery can repeatedly touch thousands of files. A planner should turn a root snapshot plus manifest state into a bounded worklist.

## Decisions
- Planner output should distinguish changed, new, missing, stale/error, and fallback-required files.
- Use indexed manifest lookups for canonical path/root/generation comparisons.
- Preserve support for explicit file and explicit root scans.
- If Merkle/directory rollups are adopted, keep them as pruning aids; per-file safety remains authoritative before skipping indexing.
- Fallback to existing bounded mtime scan when manifest state is absent, schema-stale, or internally inconsistent.

## Testing
- Synthetic roots with ~5,500 files produce worklists proportional to changed files, not total known files, after warm manifest state.
- Tests cover new file, append, rewrite with preserved mtime, delete/missing, parser error, unsupported file, symlink skip, and absent manifest fallback.
- Tests assert planner output is deterministic and deduplicated.

## Validation
- Targeted planner tests pass.
- Query plans for planner DB reads use indexes.
- Planner never schedules paths outside accepted transcript roots.
- Planner metrics make it clear how many files were discovered, skipped by manifest, scheduled, missing, and fallback-scanned.

## Files
- `src/scanner.rs`
- `src/scanner/checkpoint.rs`
- `src/scanner_tests.rs`
- `src/scanner/checkpoint_tests.rs`
- `docs/CLI.md`
- `README.md`

## Dependencies
Depends on `syslog-mcp-ssxk4.2` so persisted state exists for planner comparisons.

## References
- Current root walk/filter path: `src/scanner.rs::index_roots_with_options`.
- Current mtime overflow mitigation: `src/ai_watch.rs::run_rescan` via `index_ai_roots(... since ...)`.


LABELS: ai-indexing, performance, reliability

PARENT
  ↑ ○ syslog-mcp-ssxk4: (EPIC) Add checkpoint-backed AI transcript reconciliation planner ● P2

DEPENDS ON
  → ○ syslog-mcp-ssxk4.2: Persist manifest updates during AI transcript indexing ● P2

BLOCKS
  ← ○ syslog-mcp-ssxk4.4: Wire AI watch overflow recovery to manifest reconciliation ● P2
  ← ○ syslog-mcp-ssxk4.5: Expose manifest health in AI diagnostics ● P3

COMMENTS
  2026-06-23 19:28 Jacob Magar
    RESEARCH FINDINGS TO APPLY:
    - FACT: A persisted manifest cannot safely avoid all filesystem touching after lossy overflow without another reliable journal; directory mtimes do not change on file append.
    - ACCEPTANCE FEEDBACK: Reframe target from "no full-tree walk" to "no full unchanged-file open/hash/parse/DB rewrite after warm state." Enumeration/stat may remain O(total files), but expensive work should be O(changed + new + missing + stale/error).
    - PATTERN: Planner output should include changed/new/missing/stale/error/fallback classifications plus counters: `discovered`, `stat_checked`, `scheduled`, `hash_verified`, `missing`, `skipped_by_manifest`, and `fallback_reason`.
    - PATTERN: Directory rollups or Merkle nodes should be pruning hints only. Per-file metadata/hash remains authoritative before skipping indexing.
    - VALIDATION FEEDBACK: Query-plan tests should verify indexed SEARCH behavior without depending on exact SQLite EQP wording.
  2026-06-23 19:33 Jacob Magar
    ENGINEERING REVIEW FEEDBACK:
    - CONTRACT: Scanner owns path discovery, path safety, and reconciliation planning. Service owns async DB execution. Watcher selects manifest overflow reconciliation and bounded mtime fallback. `ai_watch.rs` must not inspect manifest tables directly.
    - OUTPUT CLASSES: Keep planner states minimal: changed, new, missing/tombstoned, stale-or-error, fallback. Treat moves/renames as missing plus new; first-class move detection is out of MVP.
    - COUNTERS: `total_files`, `stat_checked`, `scheduled`, `opened`, `hash_verified`, `parsed_records`, `db_writes`, `write_lock_ms`, `elapsed_ms`, `skipped_by_manifest`, and `fallback_reason` should be available for tests/logs.
    - SECURITY GATE: Manifest rows are advisory only; revalidate every scheduled path against current root/symlink/file-type/file-size/supported-format policy.
    - FALLBACK GATE: Manifest absent, stale, corrupt, partially migrated, or planner read-error falls back only to the existing bounded mtime scan, never an unbounded rescan.
    - QUERY GATE: Add EXPLAIN/query-plan tests for root+relative path, stale/missing anti-join, manifest_state, and generation queries.

syslog-mcp-ssxk4.4

○ syslog-mcp-ssxk4.4 · Wire AI watch overflow recovery to manifest reconciliation   [● P2 · OPEN]
Owner: Jacob Magar · Type: task
Created: 2026-06-23 · Updated: 2026-06-23

DESCRIPTION
## What
Change `cortex ai watch` overflow recovery so it uses the manifest-backed reconciliation planner when available, while keeping the current bounded/rate-limited mtime rescan as a safe fallback.

## Context
The immediate fix bounds overflow rescans to a recent lookback and rate-limits retries. This task should reduce the remaining fallback cost by scheduling only manifest-detected work after overflow or missed events.

## Decisions
- Preserve `OVERFLOW_RESCAN_LOOKBACK` and retry rate limiting as defensive backstops.
- Do not make initial scan slower or less reliable.
- Emit structured counters for planner path: manifest hits, scheduled files, fallback reason, missing files, and elapsed time.
- If manifest reconciliation fails, log a bounded warning and fall back to the existing since-based scan, not an unbounded scan.
- Keep explicit `--path` watch behavior correct for both root and single-file targets.

## Testing
- Unit tests or harness tests simulate notify overflow and assert planner-backed worklist is used after warm manifest state.
- Tests cover absent/stale manifest fallback to bounded mtime rescan.
- Tests assert repeated overflow events remain rate-limited.
- Tests cover explicit file watch and default-root watch paths.

## Validation
- `cargo test ai_watch` and relevant scanner tests pass.
- Manual/log-level verification shows overflow recovery emits bounded planner counters.
- No regression to current overflow fix from `syslog-mcp-jumyo`.

## Files
- `src/ai_watch.rs`
- `src/app/services/ai_indexing.rs`
- `src/scanner.rs`
- `src/scanner_tests.rs`
- `docs/CLI.md`

## Dependencies
Depends on `syslog-mcp-ssxk4.3` for planner API.

## References
- `src/ai_watch.rs::run_rescan`
- `src/ai_watch.rs::overflow_rescan_since`
- `src/app/services/ai_indexing.rs::index_ai_roots`


LABELS: ai-indexing, ai-watch, performance, reliability

PARENT
  ↑ ○ syslog-mcp-ssxk4: (EPIC) Add checkpoint-backed AI transcript reconciliation planner ● P2

DEPENDS ON
  → ○ syslog-mcp-ssxk4.3: Implement manifest-backed transcript reconciliation planner ● P2

BLOCKS
  ← ○ syslog-mcp-ssxk4.6: Benchmark and document AI transcript manifest reconciliation ● P3

COMMENTS
  2026-06-23 19:28 Jacob Magar
    RESEARCH FINDINGS TO APPLY:
    - FACT: Watcher overflow is lossy; Linux inotify overflow cannot tell Cortex which paths were lost. A reconciliation pass remains required.
    - PATTERN: Keep the existing 5-minute lookback and 60-second overflow retry rate-limit as safety rails.
    - PATTERN: On manifest failure, fallback must be the current bounded mtime scan, not an unbounded rescan.
    - OBSERVABILITY FEEDBACK: Watch logs/counters should prove the planner avoided unchanged-file opens/hashes/parses, not merely report fewer ingested rows.
  2026-06-23 19:33 Jacob Magar
    ENGINEERING REVIEW FEEDBACK:
    - RISK: Preserve current overflow safety rails: 5-minute lookback and 60-second retry rate-limit.
    - RISK: Notify overflow, internal channel overflow, and pending queue overflow should get separate counters/reason codes where possible; a single boolean hides the failure mode.
    - OBSERVABILITY GATE: Emit planner elapsed time plus scheduled/opened/hashed/parsed/fallback counters so logs prove unchanged files were not expensive-worked.
    - DELETE GATE: Watch remove events should mark/tombstone through manifest state where possible; broad filesystem missing verification belongs to explicit prune/repair/checkpoint commands.
    - SECURITY GATE: Overflow logs should avoid sampled path lists by default because logs become persistent syslog records.

syslog-mcp-ssxk4.5

○ syslog-mcp-ssxk4.5 · Expose manifest health in AI diagnostics   [● P3 · OPEN]
Owner: Jacob Magar · Type: task
Created: 2026-06-23 · Updated: 2026-06-23

DESCRIPTION
## What
Add operator-visible manifest health to existing AI diagnostics without requiring direct SQLite inspection or a full filesystem scan in the common case.

## Context
`cortex ai checkpoints`, `cortex ai doctor`, and API checkpoint endpoints already expose per-source state. Manifest-backed reconciliation needs observable root/generation/fallback health so future incidents can be diagnosed from Cortex itself.

## Decisions
- Prefer additive fields to existing diagnostic JSON where compatibility matters.
- Surface root count, manifest file count, stale generation count, missing/tombstoned count, last reconciliation time, last fallback reason, and oldest stale root where feasible.
- Keep `ai_watch_status` useful during DB errors: preserve OS/process probe information and return partial diagnostic context when possible.
- Do not expose raw transcript contents or overly long local paths beyond existing checkpoint behavior.

## Testing
- CLI/API tests validate JSON shape and human output for manifest health.
- DB outage tests preserve partial status rather than discarding already-collected OS probe data.
- Missing manifest / pre-migration DB tests return clear fallback status.

## Validation
- `cargo test` passes.
- `cortex ai doctor --json` and `cortex ai checkpoints --json` remain parseable.
- Docs mention new fields and their incident-response meaning.

## Files
- `src/scanner/checkpoint.rs`
- `src/app/services/ai_indexing.rs`
- CLI command modules for `cortex ai doctor/checkpoints`
- API handler/docs for `/api/ai/checkpoints` or diagnostics
- `docs/CLI.md`
- `docs/api.md`
- `README.md`

## Dependencies
Depends on `syslog-mcp-ssxk4.3` for manifest state and planner counters.

## References
- Local memory finding: `ai_watch_status` should preserve OS probe data when DB health fails.
- `src/scanner/checkpoint.rs::doctor`
- `docs/CLI.md` AI diagnostics sections.


LABELS: ai-indexing, api, diagnostics, performance, reliability

PARENT
  ↑ ○ syslog-mcp-ssxk4: (EPIC) Add checkpoint-backed AI transcript reconciliation planner ● P2

DEPENDS ON
  → ○ syslog-mcp-ssxk4.3: Implement manifest-backed transcript reconciliation planner ● P2

BLOCKS
  ← ○ syslog-mcp-ssxk4.6: Benchmark and document AI transcript manifest reconciliation ● P3

COMMENTS
  2026-06-23 19:28 Jacob Magar
    RESEARCH FINDINGS TO APPLY:
    - FACT: Current `ai doctor` missing-checkpoint detection can still iterate every checkpoint path and call `Path::exists()`.
    - PATTERN: Common manifest health should be DB-backed: root count, file count, stale generation count, missing/tombstoned count, last reconciliation time, last fallback reason, and oldest stale root where feasible.
    - SCOPE FEEDBACK: Do not redo old `ai_watch_status` resilience work wholesale; current service status already degrades DB health errors into partial status. Preserve that and add manifest fields carefully.
    - VALIDATION FEEDBACK: Reserve broad filesystem verification for explicit/checkpoint-prune/repair paths, not the fast health path.
    - DIAGNOSTIC FEEDBACK: Include DB path/runtime identity where useful to avoid repeating the prior "empty DB vs real DB" false-negative confusion.
  2026-06-23 19:33 Jacob Magar
    ENGINEERING REVIEW FEEDBACK:
    - SCOPE: Add optional manifest-health subreport; do not build a second diagnostics project.
    - FAST PATH: Common health should be DB-backed by default and avoid iterating every checkpoint with `Path::exists()`.
    - PARTIAL FAILURE: Manifest health must degrade like current `ai_watch_status`: preserve OS/process status and include bounded reason codes when DB/manifest reads fail.
    - PRIVACY: Read-token diagnostics should expose aggregate counts and bounded reason codes by default. Full paths, per-file states, inode/dev/ctime, tombstone lists, parse previews, and content-derived identifiers require admin/verbose or local-only mode.
    - CONTENT SAFETY: Manifest diagnostics must not add transcript line previews, record keys, or hashes intended to identify transcript text.

syslog-mcp-ssxk4.6

○ syslog-mcp-ssxk4.6 · Benchmark and document AI transcript manifest reconciliation   [● P3 · OPEN]
Owner: Jacob Magar · Type: task
Created: 2026-06-23 · Updated: 2026-06-23

DESCRIPTION
## What
Add performance evidence and operator documentation for manifest-backed reconciliation, including a regression guard for the ~5,500 transcript-file incident scale.

## Context
The original suspicious workload repeatedly scanned roughly 5,553 transcript files, with one `index_ai_roots` run taking 152801ms and DB lock fallout. The follow-up should prove the new manifest path reduces repeated overflow reconciliation cost after warm state.

## Decisions
- Use synthetic transcript roots for deterministic tests/benchmarks; do not rely on a developer's live transcript corpus.
- Measure warm manifest reconciliation separately from initial scan.
- Record metrics that compare total files, scheduled files, skipped-by-manifest files, elapsed time, and DB writes.
- Document fallback behavior clearly so operators know when a bounded scan is still expected.

## Testing
- Synthetic benchmark/test creates thousands of transcript-shaped files and verifies warm reconciliation schedules only changed files.
- Regression test fails if warm overflow reconciliation performs work proportional to all known files when only a small subset changed.
- Docs include implementation notes and troubleshooting guidance.

## Validation
- Benchmark or ignored perf test has clear invocation instructions.
- Normal `cargo test` remains reasonably fast.
- README/CLI docs explain initial scan vs warm reconciliation vs fallback scan behavior.

## Files
- `src/scanner_tests.rs` or a dedicated benchmark/test harness
- `docs/CLI.md`
- `README.md`
- Optional performance report doc under `docs/`

## Dependencies
Depends on `syslog-mcp-ssxk4.4` and `syslog-mcp-ssxk4.5` so behavior and diagnostics exist before documentation is finalized.

## References
- Incident evidence: repeated ~5,553-file scans, DB lock failures, one 152801ms `index_ai_roots` run.
- Prior immediate fix bead: `syslog-mcp-jumyo`.


LABELS: ai-indexing, docs, performance, reliability

PARENT
  ↑ ○ syslog-mcp-ssxk4: (EPIC) Add checkpoint-backed AI transcript reconciliation planner ● P2

DEPENDS ON
  → ○ syslog-mcp-ssxk4.4: Wire AI watch overflow recovery to manifest reconciliation ● P2
  → ○ syslog-mcp-ssxk4.5: Expose manifest health in AI diagnostics ● P3

COMMENTS
  2026-06-23 19:28 Jacob Magar
    RESEARCH FINDINGS TO APPLY:
    - FACT: Incident guardrail scale is roughly 5,553 transcript files, repeated scans, DB lock fallout, and one `index_ai_roots` run around 152801ms.
    - BENCHMARK FEEDBACK: Measure cold initial scan, warm metadata reconciliation, and warm changed-file reconciliation separately.
    - ACCEPTANCE FEEDBACK: Regression should fail if warm overflow opens/hashes/parses thousands of unchanged files; it should not fail merely because the planner enumerates/stats files.
    - METRICS FEEDBACK: Record total files, stat-checked files, scheduled files, opened files, hash-verified files, parsed records, DB writes, elapsed time, and fallback reason.
  2026-06-23 19:33 Jacob Magar
    ENGINEERING REVIEW FEEDBACK:
    - TEST STRATEGY: Keep normal `cargo test` fast with counter-based tests; put the full ~5,553-file incident-scale benchmark behind ignored/manual perf invocation if needed.
    - BENCHMARK MATRIX: Measure cold initial scan, warm no-change reconciliation, and warm small-change reconciliation separately.
    - ACCEPTANCE: Warm no-change should have zero scheduled/opened/hashed/parsed unchanged files, bounded DB writes, no fallback, and elapsed time far below the 152801ms incident run.
    - METRICS: Capture `total_files`, `stat_checked`, `scheduled`, `opened`, `hash_verified`, `parsed_records`, `db_writes`, `write_lock_ms`, `elapsed_ms`, and `fallback_reason`.
    - PRIVACY: Bench/log counters only by default; avoid sampled path lists unless admin/verbose/local-only.

Referenced Prior Mitigation Bead: syslog-mcp-jumyo

✓ syslog-mcp-jumyo [BUG] · Bound AI watch overflow rescans   [● P1 · CLOSED]
Owner: Jacob Magar · Assignee: Jacob Magar · Type: bug
Created: 2026-06-22 · Started: 2026-06-22 · Updated: 2026-06-22
Close reason: Bounded cortex-ai-watch overflow rescans to recent mtimes, rate-limited overflow retries, and hardened related timing-sensitive tests.

DESCRIPTION
cortex-ai-watch can enter repeated full transcript-root rescans after notify event overflow. This repeatedly walks thousands of transcript files, contends on SQLite, and was implicated in the dookie unresponsive incident. Optimize the fallback so overflow recovery is bounded/observable and avoids scanning every transcript every time.

---

Actual Lavra Engineering Review Applied (2026-06-25)

This section is the applied output of the real lavra-eng-review flow for this GitHub issue: architecture-strategist, code-simplicity-reviewer, security-sentinel, and performance-oracle all reviewed the issue body as the complete plan text. The issue remains self-contained: the epic bead, child beads, and prior mitigation bead are embedded above, so implementation must not require opening Beads for context.

Applied Recommendation Checklist

• Architecture: require planner DB phases to be bounded. DB reads/writes may use run_db, but directory enumeration, stat, open, hash, and parse work must happen outside SQLite write locks and must not monopolize DB worker permits.
• Architecture: model storage pressure explicitly. Track storage_preflight_blocked, storage_mid_file_blocked, and partial_import_committed; none of those states may advance a completed root generation or mark a file fresh.
• Architecture: preserve ai_watch_status behavior. Blocking process-start probing must stay off the async runtime, and OS/process fields must survive manifest/DB health failures as degraded subreports.
• Architecture: cover Gemini whole-file JSON semantics. Missing messages, skipped empty messages, parse failures, and rewrites must update manifest/checkpoint state correctly, not only JSONL append semantics.
• Architecture: tolerate partial manifest schema/migration state where practical; otherwise fall back bounded with a clear reason.
• Architecture: watch remove events may mark sources missing/tombstoned, but destructive checkpoint pruning remains explicit/admin and must not happen silently.
• Simplicity: do not create a second manifest subsystem. Extend CheckpointStore and existing transcript source metadata paths.
• Simplicity: MVP must not persist per-file generation updates for unchanged files. Use a temporary/in-memory observed set and persist only root run summaries plus changed/new/missing/stale/error transitions.
• Simplicity: reconcile zero-hash performance with preserved-mtime rewrite correctness. Trusted unchanged files with matching size/mtime/fingerprint must skip without hash; bounded hash fallback is allowed only when fingerprint evidence is absent or suspicious.
• Simplicity: no new public CLI/API/MCP surface in MVP. Add fields to existing diagnostics only.
• Performance: warm no-change planner output must not route trusted unchanged files into index_file_with_options; today that path can still open/hash before skip.
• Performance: add exact index requirements for planner paths: root/path identity, state lookup, generation/run lookup where used, and stale/missing candidate lookup. Query-plan tests must reject planner full scans.
• Performance: require benchmark/counter thresholds. Warm no-change must show near-zero scheduled/opened/hashed/parsed work and bounded DB writes/write-lock time.
• Security: manifest diagnostics default to aggregate counts and bounded enum reason codes. Full paths, per-file state, hashes/fingerprints, tombstone lists, parse previews, record keys, inode/dev/ctime require explicit admin/verbose/local-only mode per transport.
• Security: scheduled paths must reuse existing scanner path policy and revalidate canonical root containment, symlink status, regular-file type, supported format, and size immediately before every open/hash/parse.
• Security: fallback must preserve original target scope and time bound: explicit file stays explicit file, explicit root stays that root, default roots stay default roots, and never unbounded.
• Security: read-token diagnostics must not expose path lists, content hashes, record keys, inode/dev/ctime, or previews by default.
• Security: add a linked/deferred dependency advisory task for quinn-proto/audit-policy drift; do not block this planner issue unless implementation touches dependency upgrades.

Applied: 18. Skipped: 0. Deferred below: 5.

Refined Implementation Requirements

• Planner skip predicate: a trusted unchanged file is one whose canonical path is still under an accepted root, whose file type/symlink/size constraints pass immediately before any expensive work, and whose size + mtime + trusted platform fingerprint match stored metadata. Only suspicious or fingerprint-missing candidates may use a bounded hash verification path.
• Planner execution model: collect a short DB snapshot, release DB worker permits, enumerate/stat in filesystem code, then perform bounded DB write batches for changed/new/missing/stale/error transitions. Never hold a write transaction across filesystem traversal.
• Generation/run model: persist coarse root run summaries and changed states. Do not update every unchanged source row merely to mark it seen.
• Storage-block model: preflight storage block, mid-file chunk block, and partial committed import are different states. None may mark a source fresh or advance root completion.
• Delete model: remove events become missing/tombstone metadata first. Physical checkpoint pruning is explicit/admin-only.
• Diagnostics model: default health and MCP/REST read responses use aggregate counts, timings, and bounded reason enums. Full path/fingerprint/tombstone detail is admin/verbose/local-only only.
• Fallback model: corrupt/partial manifest, migration drift, DB health failure, or planner error falls back to the existing bounded mtime scan using the same target scope and since bound.
• Gemini model: whole-file JSON sessions must participate in the same skip/rewrite/error states as JSONL sources, with tests for missing messages, parse failure, skipped empty messages, and preserved-mtime rewrite detection.

Failure Modes Required In Tests

Codepath	Failure Mode	Required Rescue/Test/User/Log Behavior
Planner scheduling	Warm unchanged files reach index_file_with_options and trigger open/hash storm	Skip before indexer; counters prove opened/hashed/parsed near zero; structured counters logged
Service execution	Filesystem walk runs under run_db permit and starves MCP reads	Split DB/FS phases; mocked slow stat does not occupy DB worker; slow phase timing logged
Root bookkeeping	Warm run updates every known row	Counter/benchmark rejects per-file generation writes; user sees stable latency
Storage preflight	Storage blocked before open marks file fresh	No completed generation advance; storage-blocked count and warning logged
Storage mid-file	Committed chunk plus later block marks file complete	Partial state; repair signal; no fresh mark
Preserved-mtime rewrite	Size/mtime unchanged rewrite is missed	ctime/inode/dev or bounded hash fallback; explicit test
Manifest migration	Partial schema exists without migration row	Converge safely or bounded fallback; warning reason logged
Scheduled open	Symlink swap after planning	Revalidate immediately before open; aggregate rejected count; no path leak
Overflow fallback	Planner failure triggers default/unbounded scan	Preserve target + since; fallback_reason counter/log
Diagnostics	Read token sees paths/hashes/previews	Aggregate defaults; test proves sensitive fields absent
ai_watch_status	Manifest DB error hides process/system status	Degraded manifest subreport while OS/process fields remain visible
Gemini file	Whole-file JSON missing messages is treated as successful empty import	Manifest error/stale state, diagnostic count, warning

NOT In Scope / Deferred

• Merkle tree or directory rollups; defer until benchmarks show stat enumeration is the bottleneck.
• First-class rename/move detection; missing plus new is enough for MVP.
• Raw transcript diagnostics, tombstone browsing/export, or per-file diagnostic dumps; defer until access control/redaction is designed.
• New public CLI/API/MCP commands; additive fields on existing diagnostics are sufficient.
• Exact global SQLite lock timing instrumentation; coarse planner/import write timing is sufficient unless an existing hook makes finer timing cheap.

Completion Summary

Architecture issues applied: 6 | Simplicity: 6 | Security: 6 | Performance: 7
Critical gaps now encoded: warm no-change must bypass open/hash/parse; DB permits must not be held during filesystem planning; diagnostics must not leak transcript paths/content-derived data; partial health and storage-blocked states must not silently collapse.