You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Let the storefront recover from typos in titles — a query like `blck hedhpmnes` should still surface "Black Headphones" results. tpuf does fuzzy matching as a separate query mode (not a filter), so this is structurally a second query whose results get RRF-fused with the existing CLIP vector search.
Depends on #2 — that issue introduces the RRF fuse for BM25(title) + CLIP. This issue adds a third arm to that same fuse: tpuf Fuzzy(title). Don't start until #2's fuse plumbing exists.
Approach
Three pieces — schema flag, client-side tokenization, and a second tpuf query.
1. Mark `title` as fuzzy-able in the namespace schema.
Set the attribute schema so `title` is full-text-indexed with `fuzzy: true`. Added on namespace bootstrap, routed through `layer_client.py` (no direct tpuf — per the project's layer-only client constraint).
2. Tokenize the query ourselves.
tpuf does not tokenize Fuzzy queries — the whole string is matched as a single term. So `blck hedhpmnes` would attempt to fuzzy-match the literal 14-char string, which is useless. Split on whitespace in the indexer's `/search` handler and issue one Fuzzy query per token, applying the per-token `min_query_chars` rungs:
For `blck hedhpmnes`: `blck` (4) → distance 0, `hedhpmnes` (9) → distance 1. Open question: these rungs are too strict for the example query (`blck`→`black` is 1 edit; `hedhpmnes`→`headphones` is ~3 edits). May need a more forgiving ladder (e.g. 3–4 → 1, 5–7 → 1, 8+ → 2) — decide once we can A/B against real queries.
3. Fuse with vector results via RRF.
Plug fuzzy results into the same RRF fuse #2 builds for BM25 + CLIP. Per-token fuzzy hits can either be unioned before fusing or fused as N separate arms — pick whatever #2's primitive supports cleanly.
Files
`indexer/app/layer_client.py` — `set_namespace_schema(namespace, schema)` going through Layer; also a `fuzzy_query(...)` helper if Fuzzy uses a different endpoint shape than the existing `query_namespace`.
`indexer/tests/` — table-test the tokenizer + rung selector; integration-test against a fake LayerClient that returns canned Fuzzy results.
Schema setup probably belongs in `/index` next to `create_pipeline`, idempotent like the existing 409-tolerant pipeline create.
Out of scope
Fuzzy on `description` or review text — title is the high-value typo target.
Tuning the rung thresholds — ship the spec'd values, iterate later with telemetry.
Configurable thresholds — hard-coded constant, easy to revisit.
Risk to watch
Cost / fan-out. A 4-token query becomes 4 Fuzzy round-trips plus the CLIP vector query. Need to confirm Layer batches these or that the latency budget allows the sequential calls.
Schema PATCH idempotency. Repeated `/index` calls shouldn't error — mirror the 409-tolerance pattern in `create_pipeline`.
Layer pass-through. Verify layer-gateway proxies the Fuzzy query mode + `fuzzy` schema flag without stripping. If not, that's a layer-gateway change first.
Short queries. Tokens shorter than the lowest `min_query_chars` rung should be dropped from the fuzzy arm entirely, not sent as exact matches — otherwise a stopword-y token can dominate results.
Goal
Let the storefront recover from typos in titles — a query like `blck hedhpmnes` should still surface "Black Headphones" results. tpuf does fuzzy matching as a separate query mode (not a filter), so this is structurally a second query whose results get RRF-fused with the existing CLIP vector search.
Depends on #2 — that issue introduces the RRF fuse for BM25(title) + CLIP. This issue adds a third arm to that same fuse: tpuf Fuzzy(title). Don't start until #2's fuse plumbing exists.
Approach
Three pieces — schema flag, client-side tokenization, and a second tpuf query.
1. Mark `title` as fuzzy-able in the namespace schema.
Set the attribute schema so `title` is full-text-indexed with `fuzzy: true`. Added on namespace bootstrap, routed through `layer_client.py` (no direct tpuf — per the project's layer-only client constraint).
2. Tokenize the query ourselves.
tpuf does not tokenize Fuzzy queries — the whole string is matched as a single term. So `blck hedhpmnes` would attempt to fuzzy-match the literal 14-char string, which is useless. Split on whitespace in the indexer's `/search` handler and issue one Fuzzy query per token, applying the per-token `min_query_chars` rungs:
```json
{"max_edit_distance": [
{"min_query_chars": 3, "distance": 0},
{"min_query_chars": 6, "distance": 1}
]}
```
For `blck hedhpmnes`: `blck` (4) → distance 0, `hedhpmnes` (9) → distance 1. Open question: these rungs are too strict for the example query (`blck`→`black` is 1 edit; `hedhpmnes`→`headphones` is ~3 edits). May need a more forgiving ladder (e.g. 3–4 → 1, 5–7 → 1, 8+ → 2) — decide once we can A/B against real queries.
3. Fuse with vector results via RRF.
Plug fuzzy results into the same RRF fuse #2 builds for BM25 + CLIP. Per-token fuzzy hits can either be unioned before fusing or fused as N separate arms — pick whatever #2's primitive supports cleanly.
Files
Out of scope
Risk to watch