Skip to content

Fuzzy title search: tolerate typos via tpuf Fuzzy filter #4

Description

@hev

Goal

Let the storefront recover from typos in titles — a query like `blck hedhpmnes` should still surface "Black Headphones" results. tpuf does fuzzy matching as a separate query mode (not a filter), so this is structurally a second query whose results get RRF-fused with the existing CLIP vector search.

Depends on #2 — that issue introduces the RRF fuse for BM25(title) + CLIP. This issue adds a third arm to that same fuse: tpuf Fuzzy(title). Don't start until #2's fuse plumbing exists.

Approach

Three pieces — schema flag, client-side tokenization, and a second tpuf query.

1. Mark `title` as fuzzy-able in the namespace schema.

Set the attribute schema so `title` is full-text-indexed with `fuzzy: true`. Added on namespace bootstrap, routed through `layer_client.py` (no direct tpuf — per the project's layer-only client constraint).

2. Tokenize the query ourselves.

tpuf does not tokenize Fuzzy queries — the whole string is matched as a single term. So `blck hedhpmnes` would attempt to fuzzy-match the literal 14-char string, which is useless. Split on whitespace in the indexer's `/search` handler and issue one Fuzzy query per token, applying the per-token `min_query_chars` rungs:

```json
{"max_edit_distance": [
{"min_query_chars": 3, "distance": 0},
{"min_query_chars": 6, "distance": 1}
]}
```

For `blck hedhpmnes`: `blck` (4) → distance 0, `hedhpmnes` (9) → distance 1. Open question: these rungs are too strict for the example query (`blck`→`black` is 1 edit; `hedhpmnes`→`headphones` is ~3 edits). May need a more forgiving ladder (e.g. 3–4 → 1, 5–7 → 1, 8+ → 2) — decide once we can A/B against real queries.

3. Fuse with vector results via RRF.

Plug fuzzy results into the same RRF fuse #2 builds for BM25 + CLIP. Per-token fuzzy hits can either be unioned before fusing or fused as N separate arms — pick whatever #2's primitive supports cleanly.

Files

  • `indexer/app/layer_client.py` — `set_namespace_schema(namespace, schema)` going through Layer; also a `fuzzy_query(...)` helper if Fuzzy uses a different endpoint shape than the existing `query_namespace`.
  • `indexer/app/main.py` — in `/search`, tokenize `body.query`, build per-token Fuzzy queries, hand to the RRF fuser from Hybrid product search: RRF fuse BM25(title) + CLIP vector #2.
  • `indexer/tests/` — table-test the tokenizer + rung selector; integration-test against a fake LayerClient that returns canned Fuzzy results.
  • Schema setup probably belongs in `/index` next to `create_pipeline`, idempotent like the existing 409-tolerant pipeline create.

Out of scope

  • Fuzzy on `description` or review text — title is the high-value typo target.
  • Tuning the rung thresholds — ship the spec'd values, iterate later with telemetry.
  • Configurable thresholds — hard-coded constant, easy to revisit.

Risk to watch

  • Cost / fan-out. A 4-token query becomes 4 Fuzzy round-trips plus the CLIP vector query. Need to confirm Layer batches these or that the latency budget allows the sequential calls.
  • Schema PATCH idempotency. Repeated `/index` calls shouldn't error — mirror the 409-tolerance pattern in `create_pipeline`.
  • Layer pass-through. Verify layer-gateway proxies the Fuzzy query mode + `fuzzy` schema flag without stripping. If not, that's a layer-gateway change first.
  • Short queries. Tokens shorter than the lowest `min_query_chars` rung should be dropped from the fuzzy arm entirely, not sent as exact matches — otherwise a stopword-y token can dominate results.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions