Summary
The amazon-products namespace is created without an explicit schema, so it
inherits turbopuffer's default where every attribute is filterable. We only
ever query on a tiny subset, so the index carries filterable/sortable structures
for a pile of attributes nothing reads. Trimming the schema to what we actually
use should improve write throughput and query performance noticeably.
YAGNI on filters: declare the minimal schema and let the gateway own its
managed columns.
Evidence
The pipeline (and therefore the namespace) is created here with no schema —
only id, target_namespace, distance_metric:
indexer/app.py:151 / indexer/app.py:158 — ensure_pipeline(CreatePipelineRequest(...))
common/hev_shop_common/models.py → CreatePipelineRequest has no schema field
Per-product attributes written on every vector (product_vector_attributes,
common/hev_shop_common/records.py:63):
asin, title, category, description, image_url, catalog_run_id,
avg_rating_txt, rating_cnt_txt
All of these land filterable by default, plus the gateway-managed
_hevlayer_shard and _hevlayer_upserted_at.
(Couldn't dump the live schema for this issue — GET /v1/namespaces/amazon-products/schema
needs a gateway Bearer key. Add the actual schema dump when picking this up.)
Desired schema (what we actually use)
| Attribute |
Index |
Why |
category |
filterable |
["category","Eq",...] in /search + /recommend; on-demand snapshot for the facet list (search/app.py:350,465,541) |
title |
full-text search (FTS) |
text search surface |
_hevlayer_shard |
gateway-managed |
app must not declare it; gateway sets this automatically |
_hevlayer_upserted_at |
gateway-managed |
ditto |
everything else (asin, description, image_url, catalog_run_id, avg_rating_txt, rating_cnt_txt) |
not filterable |
nothing queries them; asin mirrors the doc id, which is filterable regardless |
Fix point
Layer-only client rule applies — no direct tpuf access. The SDK already exposes
the knob:
update_turbopuffer_namespace_schema(namespace, TurbopufferSchema) (hevlayer/client.py:808)
get_turbopuffer_namespace_schema(...) to read it back
Declare the minimal schema alongside ensure_pipeline in indexer/app.py
(create the pipeline, then apply the schema). If the SDK/YAML round-trip should
carry schema on CreatePipelineRequest instead, that's a Layer-side question
worth raising — keep them in sync per CLAUDE.md's one-schema rule.
Must reconcile before shipping
/search filters on catalog_run_id when the caller passes it
(search/app.py:352). Making catalog_run_id non-filterable breaks that path.
Decide as part of the fix: drop the catalog_run_id filter as YAGNI, or keep
catalog_run_id filterable. The proposed schema above assumes we drop it.
Acceptance
Summary
The
amazon-productsnamespace is created without an explicit schema, so itinherits turbopuffer's default where every attribute is filterable. We only
ever query on a tiny subset, so the index carries filterable/sortable structures
for a pile of attributes nothing reads. Trimming the schema to what we actually
use should improve write throughput and query performance noticeably.
YAGNI on filters: declare the minimal schema and let the gateway own its
managed columns.
Evidence
The pipeline (and therefore the namespace) is created here with no schema —
only
id,target_namespace,distance_metric:indexer/app.py:151/indexer/app.py:158—ensure_pipeline(CreatePipelineRequest(...))common/hev_shop_common/models.py→CreatePipelineRequesthas noschemafieldPer-product attributes written on every vector (
product_vector_attributes,common/hev_shop_common/records.py:63):asin,title,category,description,image_url,catalog_run_id,avg_rating_txt,rating_cnt_txtAll of these land filterable by default, plus the gateway-managed
_hevlayer_shardand_hevlayer_upserted_at.(Couldn't dump the live schema for this issue —
GET /v1/namespaces/amazon-products/schemaneeds a gateway Bearer key. Add the actual schema dump when picking this up.)
Desired schema (what we actually use)
category["category","Eq",...]in/search+/recommend; on-demand snapshot for the facet list (search/app.py:350,465,541)title_hevlayer_shard_hevlayer_upserted_atasin,description,image_url,catalog_run_id,avg_rating_txt,rating_cnt_txt)asinmirrors the docid, which is filterable regardlessFix point
Layer-only client rule applies — no direct tpuf access. The SDK already exposes
the knob:
update_turbopuffer_namespace_schema(namespace, TurbopufferSchema)(hevlayer/client.py:808)get_turbopuffer_namespace_schema(...)to read it backDeclare the minimal schema alongside
ensure_pipelineinindexer/app.py(create the pipeline, then apply the schema). If the SDK/YAML round-trip should
carry schema on
CreatePipelineRequestinstead, that's a Layer-side questionworth raising — keep them in sync per CLAUDE.md's one-schema rule.
Must reconcile before shipping
/searchfilters oncatalog_run_idwhen the caller passes it(
search/app.py:352). Makingcatalog_run_idnon-filterable breaks that path.Decide as part of the fix: drop the
catalog_run_idfilter as YAGNI, or keepcatalog_run_idfilterable. The proposed schema above assumes we drop it.Acceptance
amazon-productsschema declares onlycategory(filterable) +title(FTS)_hevlayer_shard/_hevlayer_upserted_atcatalog_run_idfilter reconciled (dropped or kept filterable, decided explicitly)