Skip to content

amazon-products namespace is maximally filterable — declare a minimal schema (YAGNI on filters) #9

Description

@hev

Summary

The amazon-products namespace is created without an explicit schema, so it
inherits turbopuffer's default where every attribute is filterable. We only
ever query on a tiny subset, so the index carries filterable/sortable structures
for a pile of attributes nothing reads. Trimming the schema to what we actually
use should improve write throughput and query performance noticeably.

YAGNI on filters: declare the minimal schema and let the gateway own its
managed columns.

Evidence

The pipeline (and therefore the namespace) is created here with no schema —
only id, target_namespace, distance_metric:

  • indexer/app.py:151 / indexer/app.py:158ensure_pipeline(CreatePipelineRequest(...))
  • common/hev_shop_common/models.pyCreatePipelineRequest has no schema field

Per-product attributes written on every vector (product_vector_attributes,
common/hev_shop_common/records.py:63):

asin, title, category, description, image_url, catalog_run_id,
avg_rating_txt, rating_cnt_txt

All of these land filterable by default, plus the gateway-managed
_hevlayer_shard and _hevlayer_upserted_at.

(Couldn't dump the live schema for this issue — GET /v1/namespaces/amazon-products/schema
needs a gateway Bearer key. Add the actual schema dump when picking this up.)

Desired schema (what we actually use)

Attribute Index Why
category filterable ["category","Eq",...] in /search + /recommend; on-demand snapshot for the facet list (search/app.py:350,465,541)
title full-text search (FTS) text search surface
_hevlayer_shard gateway-managed app must not declare it; gateway sets this automatically
_hevlayer_upserted_at gateway-managed ditto
everything else (asin, description, image_url, catalog_run_id, avg_rating_txt, rating_cnt_txt) not filterable nothing queries them; asin mirrors the doc id, which is filterable regardless

Fix point

Layer-only client rule applies — no direct tpuf access. The SDK already exposes
the knob:

  • update_turbopuffer_namespace_schema(namespace, TurbopufferSchema) (hevlayer/client.py:808)
  • get_turbopuffer_namespace_schema(...) to read it back

Declare the minimal schema alongside ensure_pipeline in indexer/app.py
(create the pipeline, then apply the schema). If the SDK/YAML round-trip should
carry schema on CreatePipelineRequest instead, that's a Layer-side question
worth raising — keep them in sync per CLAUDE.md's one-schema rule.

Must reconcile before shipping

/search filters on catalog_run_id when the caller passes it
(search/app.py:352). Making catalog_run_id non-filterable breaks that path.
Decide as part of the fix: drop the catalog_run_id filter as YAGNI, or keep
catalog_run_id filterable. The proposed schema above assumes we drop it.

Acceptance

  • amazon-products schema declares only category (filterable) + title (FTS)
  • App does not declare _hevlayer_shard / _hevlayer_upserted_at
  • catalog_run_id filter reconciled (dropped or kept filterable, decided explicitly)
  • Schema applied via the Layer SDK, not direct tpuf
  • Before/after perf noted (write throughput + query latency)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions