curaitor: Recycle TSV index fast-path (partial impl of PR #17 design)

[jdidion]

Note: This MR was largely generated by Claude and has not been completely reviewed by me (the human). You should feel free to defer your review until this warning has been removed.

Summary

Partial implementation of curaitor PR #17 (Recycle Storage Scale Plan) — specifically the Obsidian-side TSV index + fast-path dedup. Skips the monthly-partitioning and SQLite-backend work because:

• Current Recycle.md is 370 lines. 1175 unique URLs across live + archives. Still well below the 5K trigger the design doc set.
• Monthly partitioning is already done by scripts/recycle-rollover.py; no new partitioning work needed.
• SQLite-backend changes belong in the curaitor TypeScript app, not the claude-plugins Python scripts.

This PR validates the fast-path design on real data so it's ready to pay off when the log grows.

What shipped

scripts/recycle-reindex.py (new)

One-shot TSV builder. Scans Curaitor/Recycle.md + the most-recent N monthly archives, extracts unique normalized URLs, writes .curaitor/recycle-index.tsv with format:

# recycle-index v1 checksum=<sha256-over-source-files>
url_normalized	source_file	title
genomeweb.com/genomic-research/ssri-approach...	Recycle.md	Genotype-Guided SSRI...
arxiv.org/abs/2404.12345	Recycle-2026-04.md	Some arxiv paper
...

Dedups across files (same URL in live Recycle.md AND an archive → one row). Header line embeds a SHA-256 checksum over the source markdown bytes so consumers can detect drift.

Idempotent, safe to re-run, atomic write via tmp + rename. Supports --dry-run and --json modes for observability.

scripts/triage-write.py — fast-path integration

• New _load_recycle_tsv(vault): reads the TSV if its checksum matches the current Recycle.md + archives. Returns a set[str] of normalized URLs on success, None on any failure (missing file / stale checksum / malformed).
• build_recycle_index(vault): prefers the TSV; falls back to line-by-line markdown parse on miss. On a miss, kicks off a background rebuild via Popen (fire-and-forget, start_new_session=True) so the next call hits the fast path.
• build_url_index(vault): reuses the cached TSV for the recycle portion; only runs markdown parse for recycle sources when the TSV is unavailable.

The fallback is the current line-by-line parse. Every code path remains correct without the TSV; the TSV is pure optimization.

Measured performance

Benchmark on synthetic Recycle.md files (local disk):

N rows	Fast path	Slow path	Speedup
1,000	0.6ms	2.3ms	3.8×
5,000	1.4ms	11.4ms	8.1×
20,000	4.0ms	44.4ms	11.1×

On the live Google Drive-backed vault (1175 URLs across 3 files), fast path is currently ~3× slower than the fallback (20.8ms vs 6.9ms) because the checksum computation reads all 3 source files and Google Drive's filesystem latency dominates at this scale. This inverts past ~3-5K rows where the per-row parse cost in the fallback grows linearly while the TSV's per-row lookup stays constant. The design doc's "<1ms p99 on indexed path" holds on local filesystems; Google Drive adds a filesystem-latency constant that benchmarks need to account for.

Parity verified: on the real vault (1175 URLs), fast path and slow path return exactly the same set with zero diff.

What's NOT in this PR (deferred)

• Monthly-partition strategy from the design doc — already done by scripts/recycle-rollover.py. No work needed.
• SQLite backend schema additions (url_normalized, tag columns + indexes) — belongs in the curaitor TypeScript app, not here.
• StorageBackend interface additions (hasRecycled(url), loadRecycleRecent(limit)) — TypeScript app, not here.
• SQLite export CLI (npm run export:recycle-md) — TypeScript app.
• Cron-wrapper auto-reindex pre-triage — not needed; the background rebuild on miss converges within one triage cycle.

Initial TSV build

The user's vault already has .curaitor/recycle-index.tsv as of this PR commit (built during spike verification). Future cron runs pick it up automatically.

For any other install: run python3 scripts/recycle-reindex.py once. Or just run any triage — the miss-handler will auto-rebuild.

Test plan

• Next triage cron fire hits the TSV fast-path (verify via timing in triage-write.py output if we add instrumentation later, or trust the parity check).
• Hand-edit Recycle.md → next triage hits checksum-mismatch → fallback runs → background rebuild → next-next triage is fast again. (Hand-test once Gemma is re-enabled, since cron is currently slow-path-only when local_triage is off.)

🤖 Generated with Claude Code