Load tester: simulate sustained, ramp, and burst traffic across hev-shop surfaces

[hev]

Goal

A repeatable load tester we can point at the live hev-shop stack to generate
realistic, mixed-workload traffic across the storefront, indexer API, and
layer-gateway. Goals:

1. Prove the public path (ALB -> web/api/gateway) holds up under expected
   demo traffic.
2. Surface the knee of each surface (where p95 latency or error rate breaks)
   so we can size HPA + cache budgets intentionally instead of guessing.
3. Give us a way to reproduce a burst before a demo and watch autoscaling +
   cold-cache behavior in dashboards.

Implementation deferred — this issue captures requirements and shape so the
work can be picked up cleanly.

Tool

k6. JS-scriptable scenarios, built-in ramping/constant-arrival
executors, clean summary + Prometheus remote-write output, runs anywhere
(laptop, CI, or a one-shot pod in the cluster). No new language in the repo
since the scripts live alongside ops tooling, not inside indexer/app/.

Repo layout

loadtest/
  README.md              # how to run locally and against prod
  scenarios/
    indexer-search.js    # POST /search mixed query corpus
    indexer-reviews.js   # GET /search/reviews mixed asin+q
    indexer-product.js   # GET /product/{asin} ASIN sampler
    indexer-meta.js      # GET /meta + /healthz cheap baseline
    storefront.js        # GET / , /product/{asin} on hev-shop.com
    gateway.js           # GET /v2/pipelines , /v2/namespaces/.../metadata
    mixed.js             # weighted blend of the above (demo profile)
  data/
    queries.json         # ~500 realistic search queries
    asins.json           # ~1k ASINs known to exist in amazon-products
  lib/
    targets.js           # base URLs from env (LOCAL / PROD)
    metrics.js           # custom trends + thresholds

ASIN + query corpora are generated by a small script that pulls from the
indexer (/meta for namespace stats, a sampling query for ASINs). The
generator is checked in so the corpus is reproducible.

Surfaces and endpoints

Indexer API (https://api.hev-shop.com):

• GET /healthz — baseline liveness, near-zero cost
• GET /meta — cached, exercises the cache path
• GET /product/{asin} — single-doc gateway fetch
• POST /search body {query, top_k} — CLIP text embed + tpuf query
• GET /search/reviews?asin=&q=&top_k= — Qwen embed + shard fan-out
  (gated by api_review_search_enabled; scenario must check /meta first
  and skip if disabled on the target pod set)
• GET /reviews/samples — read-only sampler

Storefront (https://hev-shop.com):

• GET / — homepage render
• GET /product/{asin} — detail page; exercises web -> in-cluster API
• GET /product/{asin}/image — once [[image-blob-cache]] (Image blob cache: serve product JPEGs from Aerospike via layer-gateway #5) lands

Layer gateway (https://aws-us-east-1.hevlayer.com):

• GET /v2/pipelines
• GET /v2/namespaces/amazon-products/metadata

Load profiles

Each scenario file exposes three k6 scenarios{} entries, selectable via
PROFILE env:

1. baseline — constant-arrival-rate, ~10 RPS per surface for 10 min.
   Establishes a steady-state latency floor; safe to run against prod.
2. ramp — ramping-arrival-rate, 1 -> 200 RPS over 20 min then 10 min
   soak at the top. Finds the knee and surfaces slow leaks (memory, FD,
   connection pool).
3. burst — constant-arrival-rate at 5 RPS background with three
   60-second spikes to 300 RPS spaced 3 min apart. Exercises HPA reaction,
   cold-cache penalty, and queue depth.

mixed.js runs all surface scripts concurrently under the selected profile
with weights roughly matching the demo story (search-heavy, product-heavy
tail, light gateway probes).

Thresholds (initial, tunable)

Per-scenario k6 thresholds{}:

• http_req_failed: rate<0.01 (baseline), rate<0.05 (ramp/burst)
• http_req_duration{endpoint:search}: p(95)<800 baseline, p(95)<2000 ramp
• http_req_duration{endpoint:reviews}: p(95)<1500 baseline (Qwen heavier)
• http_req_duration{endpoint:product}: p(95)<300 baseline
• http_req_duration{endpoint:healthz}: p(95)<100 always

Thresholds are advisory in ramp/burst — the goal there is to find the
break, not to fail the run on the first 500.

How we run it

Local (laptop -> prod public URLs):

cd loadtest
TARGET=prod PROFILE=baseline k6 run scenarios/mixed.js
TARGET=prod PROFILE=burst    k6 run scenarios/indexer-search.js

In-cluster (skips the ALB, isolates the app):

kubectl run k6 -n hev-shop --rm -it --restart=Never \
  --image=grafana/k6:latest \
  --env=TARGET=incluster --env=PROFILE=ramp \
  -- run - < loadtest/scenarios/mixed.js

targets.js swaps base URLs: prod uses the public DNS from AGENTS.md;
incluster uses http://hev-shop-api.hev-shop.svc.cluster.local:8080 and
the layer-gateway in-cluster service name.

Observability

• k6 stdout summary is the per-run artifact (committed to the PR comment
  when run from CI).
• Stream metrics via --out experimental-prometheus-rw to the cluster's
  Prometheus so we can correlate against pod CPU/memory and the
  layer_perf panels.
• Tag every request with {scenario, endpoint, profile} so grafana panels
  can slice by surface.

Guardrails

• Never run ramp or burst against prod without coordinating — the
  amazon-products namespace is shared with other demos and the
  layer-gateway is a real service. README must call this out.
• No write endpoints in scope. /index, /backfill, and any gateway PUT
  routes are explicitly excluded — load tester is read-only.
• Respect api_review_search_enabled; CPU-only pods will 503 on
  /search/reviews and that's expected, not a failure to record.

Out of scope

• Chaos / fault injection (separate concern; would belong in a different
  tool).
• Synthetic monitoring / uptime probes (these are continuous, low-rate,
  alerting concerns — different shape).
• Frontend rendering perf (Lighthouse / WebPageTest territory).
• Load-testing the extraction pipeline workers — they're not request/response
  shaped; backpressure there is measured via queue depth, not RPS.

Done when

• loadtest/ directory exists with the scenarios above and a README that
  documents TARGET x PROFILE invocations.
• A baseline run against prod completes green and the summary is attached
  to the PR.
• A ramp run (against a scratch namespace or off-peak) produces a
  knee-of-the-curve chart we can paste into a follow-up sizing issue.