feat: Jaccard token similarity as post-RRF reranking signal

[ajianaz]

Summary

Add Jaccard token overlap as an orthogonal recall reranking signal (#719).

Problem

Current recall uses two signals: BM25 (FTS5, IDF-weighted) and vector cosine (semantic embedding). Both miss a simple but effective signal: raw token overlap. A query "rust memory engine" and content "rust is a fast memory engine for AI" have 3/8 token overlap — but BM25 may downweight common terms while cosine may miss exact keyword matches.

Hermes holographic memory uses Jaccard as a 0.3-weighted signal in its hybrid fusion (alongside 0.4 FTS + 0.3 HRR).

Changes

New: crates/uteke-core/src/jaccard.rs

• tokenize(text) → lowercase HashSet, strips punctuation
• jaccard_similarity(query, content) → |A∩B| / |A∪B| in [0, 1]
• 9 unit tests covering edge cases

Modified: crates/uteke-core/src/operations.rs

• Post-RRF Jaccard boost when jaccard_weight > 0.0
• Tokenizes content + tags for richer overlap
• Re-sorts after boost

Modified: crates/uteke-core/src/lib.rs

• Uteke struct: new jaccard_weight: f32 field (default 0.0)
• set_jaccard_weight() setter with 0.0-1.0 clamp

Modified: crates/uteke-cli/src/{config.rs, main.rs}

• [recall].jaccard_weight in uteke.toml (default 0.0 = off)
• Applied on startup

Configuration

[recall]
jaccard_weight = 0.1  # 0=off, 0.10-0.15 recommended

Trade-offs

Pro	Con
Catches exact keyword matches BM25/cosine miss	Extra tokenization per query
Zero cost when weight=0.0 (default)	Bag-of-words: ignores word order
Tags boost: tag tokens included in content set	May over-boost short content

Closes #719

[github-actions]

🔍 Cora AI Code Review

⚠️ Review recommended — warnings found.

🟡 Warning (1)

• unknown:? — Cora AI review could not complete: failed to parse LLM JSON response: parse failed (original: EOF while parsing a value at line 1 column 0, after repair: EOF while parsing a value at line 1 column 0). Review was skipped; no code quality issues were found by the automated check.

---

Review powered by cora-cli · BYOK · MIT