feat(store): mem_search returns 0 when no observation contains all query tokens

[Basparin]

📋 Pre-flight Checks

• I have searched existing issues and this is not a duplicate
• I understand this issue needs status:approved before a PR can be opened

🔍 Problem Description

mem_search returns 0 results for multi-token queries when no single observation contains every token, even if observations exist for each token individually. The failure is silent: callers see "no results" and conclude the data is missing, sometimes saving duplicates of memos that already exist.

💡 Proposed Solution

Add a match_mode option ("all" | "any") to Store.Search() and propagate to the mem_search MCP tool. The OR branch can reuse the join-with-OR logic already present in sanitizeFTSCandidates (internal/store/relations.go), which FindCandidates uses internally for the same kind of relaxed match.

Example usage:

mem_search(query: "auth compliance session", match_mode: "any")

📦 Affected Area

Store (database, queries)

🔄 Alternatives Considered

• Auto-relax fallback: retry as OR when AND returns 0, with a flag in the response so callers can detect the relaxation. Hides the param at API level.
• Client-side query validation: fragile, every consumer must implement and maintain it independently.
• Related work: feat(store): trigram + fuzzy search with weighted confidence scores #257 (trigram + fuzzy) targets a different failure mode (typo/substring); Proposal: Five-Layer Memory Enhancement (Semantic Search, Salience, Graph, Consolidation, Priming) #168 (semantic memory enhancement) is broader scope (embeddings).

📎 Additional Context

Synthetic repro (fresh database, observations with partial overlap):

mem_save(title: "Auth session middleware", content: "")
mem_save(title: "Compliance audit notes",   content: "session policy")
mem_save(title: "OAuth tokens",              content: "auth and compliance")

mem_search("auth")        -> 2 hits
mem_search("compliance")  -> 2 hits
mem_search("session")     -> 2 hits
mem_search("auth compliance session") -> 0 hits

Each observation contains 2 of the 3 query tokens; the AND query still drops everything because no single observation contains all three. Root cause at internal/store/store.go:5874-5884: sanitizeFTS() wraps each token in quotes joined by spaces, and FTS5 reads spaces between quoted tokens as implicit AND.

Stopgap mitigation (until match_mode lands)

A minimal client-side pre-tool advisory catches the silent collapse before it hits the store. Wired as a Claude Code PreToolUse hook on mcp__plugin_engram_engram__mem_search:

import sys, json, re
data = json.load(sys.stdin)
if data.get("tool_name", "").endswith("__mem_search"):
    q = (data.get("tool_input") or {}).get("query") or ""
    tokens = re.findall(r"[A-Za-z0-9_-]{2,}", q)
    if len(tokens) >= 6:  # placeholder threshold; tune locally
        sys.stderr.write(
            f"engram-query: {len(tokens)} tokens; FTS5 AND likely returns 0. "
            f"Consider splitting into shorter queries.\n"
        )
sys.exit(0)  # advisory only, never blocks

The hook prints a stderr warning when a query is unlikely to match under AND. It does not block the call. Useful while the design is decided so silent zero-results stop being silent.

[Basparin]

Sharing field evidence in case useful for the triage decision on this
issue. Aware we're inside the 2-day triage SLA — this is data, not a ping.

This complements two adjacent issues that document related angles of the
same recall problem:

• Add semantic search layer on top of FTS5 to improve mem_search recall #233 — semantic search proposal (recall via embeddings)
• feat(store): replace FTS5 default ranking with weighted BM25 #241 — weighted BM25 ranking (precision)

The three form a complementary stack: syntactic relaxation (#352) +
ranking precision (#241) + semantic recall (#233). Each addresses a
distinct failure mode without overlapping scope.

Local installation across 2 active projects (anonymized as project-a
and project-b), 149 queries logged by a client-side PostToolUse hook
on mem_search over ~6 days (2026-05-06 → 2026-05-12).

Specific query strings are anonymized in this comment to avoid leaking
domain context. Full anonymized log + concrete query pairs available on
request.

1. Zero-hit rate scales monotonically with token count

Tokens	Queries	Zero-hit rate
2	27	11%
3	35	31%
4	40	35%
5	32	50%
6	10	50%

Each additional AND-strict token adds ~10-15pp to the zero-hit rate.
By 5 tokens, half of all queries return nothing.

Global zero-hit rate: 51/149 = 34%.

2. AND-collapse confirmed by accidental natural experiment

Three queries from the same session, ~30s apart, target a single domain
boundary:

Query	Tokens	Hits
<concept-A>	3	2
<concept-B>	2	5
<concept-A> <concept-B>	5	0

Both component topics have observations (2 and 5 respectively). Their
AND-intersection has none. The combined query collapses silently. This
matches the pattern documented in #233 update (2026-04-29) Section A
with a larger corpus.

3. Client-side discipline doesn't fix it

A SessionStart hook reminds the agent of FTS5 AND-strict semantics every
session. Agents do learn to prefer 2-4 token queries early on. But:

• First 30 queries: 37% zero-hit
• Last 30 queries: 37% zero-hit

No meaningful temporal improvement. As context grows mid-session, queries
naturally drift toward 5-6 tokens again (richer concepts to search), and
the zero-hit rate stays constant. Behavioral guidance helps starting
state, not steady state.

4. Other natural pairs in log

Four additional pairs targeting the same concept with different token
combinations. Adding or substituting tokens for the same concept flips
hits to zero, even at the same token count:

Pair	Working query	Tokens	Hits	Failed query (same concept)	Tokens	Hits
1	<X-a>	4	4	<X-b>	5	0
2	<Y-a>	3	3	<Y-b>	4	0
3	<Z-a>	3	4	<Z-b>	4	0
4	<W-a>	4	2	<W-b>	4	0

The information exists; AND-strict hides it on the failing variant.

Offer

If useful for triage or testing:

• Full anonymized log available on request (149 entries, JSONL)
• Concrete query pairs reproducible against any engram instance with
  overlapping topic_keys
• Willing to contribute test cases once status:approved lands