Image blob cache: serve product JPEGs from Aerospike via layer-gateway

[hev]

Goal

Serve product images out of the Aerospike doc cache so the storefront renders product thumbnails and detail-page hero images with ~2 ms backend lookup instead of an Amazon CDN round-trip. Same write that produces a CLIP image vector also captures the JPEG bytes — one ingest, one disposition.

Depends on

docs/layer-team-image-blob-cache-2026-05-19.md — layer-gateway change: typed blobs field on the doc model, native Value::Blob in Aerospike, dedicated GET /v2/namespaces/{ns}/documents/{id}/blobs/{key} route, blobs explicitly not mirrored to turbopuffer. This issue assumes that shape lands; do not start hev-shop work until it does.

Doc model on the hev-shop side

When embed-products upserts a product, include the JPEG bytes alongside the existing vector + attributes:

{
  "id": "B00FI7TCGI",
  "vector": [...],
  "attributes": { "asin": "...", "title": "...", "image_url": "...", ... },
  "blobs": {
    "image": { "data": "<base64 jpeg>", "content_type": "image/jpeg" }
  }
}

image_url stays in attributes as the durable source-of-truth pointer and as the web-side fallback. blob:image is the fast path.

Pipeline changes

embed-products (modify) — indexer/app/pipeline.py:172

The image is already on local disk at meta["image_path"] (line 198) before we call the embedder. After the embed succeeds, image.read_bytes() and include in the upsert payload. Hard cap at 512 KB; if larger, skip the blob (still write the vector) and emit blob_oversize_total.

• Add a _load_blob(path) helper returning (bytes, content_type) — sniff from Path.suffix (.jpg/.jpeg → image/jpeg, .png → image/png, .webp → image/webp).
• Failure mode: if the bytes can't be read, log + increment blob_read_failed_total and upsert without blobs. The vector write still wins; the web falls back to image_url.

backfill-images (new stage) — populates blobs for products already in the index.

• pipeline.py STAGES entry: pipeline_attr=\"backfill_images_pipeline_id\", from_stage=\"pending\", claim size knob, no setup.
• Processor fetches the product attributes, re-fetches the image from image_url (we don't keep the staged JPEGs around), writes via the existing layer-client patch path with blobs only — no vector touched.
• WORKER_TYPE=\"backfill-images\" → wire into worker.STAGE_FOR_WORKER_TYPE.
• Run once across the existing ~1.5M-row namespace; can run with low parallelism since it's bandwidth-bound on Amazon CDN, not CPU.

layer_client.py — indexer/app/layer_client.py

• Extend upsert_documents() to accept a blobs argument that flows into the request JSON.
• New fetch_blob_url(namespace, doc_id, key) helper that returns the gateway URL for the dedicated blob route — used by the API to build redirect targets (see below).

API changes (hev-shop-api)

indexer/app/main.py — add one route, a thin redirect:

GET /product/{asin}/image
  → 302 to https://aws-us-east-1.hevlayer.com/v2/namespaces/amazon-products/documents/{asin}/blobs/image
  → 302 to the original image_url (from attributes) if blob missing / gateway 404
  Cache-Control: public, max-age=300 on the redirect itself

302 over proxying keeps the indexer API out of the byte path. Browser hits the gateway directly on the second request and follows the long-lived Cache-Control the gateway sets.

Alternative we considered and rejected: server-side proxy that streams bytes through hev-shop-api. Adds a hop + memory pressure for no win once the browser cache is warm. Revisit only if we hit CORS or auth issues with the direct gateway URL from a browser.

Web changes

web/lib/backend.ts (and the two render sites in web/app/page.tsx:128 + web/app/product/[asin]/page.tsx:140): use ${API_BASE}/product/${asin}/image as the src instead of image_url. Next/Image loader config may need updating so it doesn't try to optimize the redirect target.

Metrics

• hev_shop_blob_write_total{result=\"ok|read_failed|oversize\"} (indexer)
• hev_shop_blob_fetch_failed_total (indexer, backfill stage, Amazon CDN miss)
• Gateway-side blob fetch latency lands in the existing layer_perf we already surface in the UI — no new wiring.

Files

• indexer/app/records.py:173 — keep image_url + image_path as-is; no schema change.
• indexer/app/pipeline.py:172 — extend embed-products to read bytes after embed.
• indexer/app/pipeline.py:688 — new STAGES entry for backfill-images.
• indexer/app/worker.py:16 — map WORKER_TYPE=\"backfill-images\" → new stage.
• indexer/app/layer_client.py — blobs arg on upsert; new fetch_blob_url helper.
• indexer/app/config.py — backfill_images_pipeline_id, blob_max_bytes (default 524288).
• indexer/app/main.py — GET /product/{asin}/image redirect handler.
• helm/hev-shop/templates/ — new worker Deployment for backfill-images (model on existing embed-products worker).
• web/lib/backend.ts + web/app/page.tsx:128 + web/app/product/[asin]/page.tsx:140 — switch image src.
• indexer/tests/test_pipeline.py — blob round-trip in embed-products (small fixture JPEG), oversize-skip path, backfill stage processor.

Acceptance

• curl -sI https://api.hev-shop.com/product/B00FI7TCGI/image returns 302 to the gateway blob URL.
• Following the redirect returns image/jpeg bytes with Cache-Control: public, immutable and a backend timing under 10 ms p95 (the 2 ms target is for the Aerospike hop itself; total includes ALB + TLS).
• Storefront thumbnails on https://hev-shop.com and the product detail page render from the new route, fall back to image_url cleanly if the blob endpoint 404s.
• backfill-images stage completes across amazon-products namespace; hev_shop_blob_write_total{result=\"ok\"} matches indexed product count within 5%.
• Existing embed-products throughput stays within 10% of pre-change baseline (the extra read_bytes() is cheap but worth confirming).

🤖 Generated with Claude Code