[design·perf] Response-time budget by route — separate real-user latency from QA/test overhead

[atomchung]

Status

Discussion / measurement-first — M1 product-quality risk, not an active implementation front.

Owner ruling, 2026-07-30 and expanded 2026-07-31: every answer the product gives must arrive within a bounded, predictable time. The early experience of waiting 10–20 minutes for one review card is the anti-goal. The first budget must come from live measurement rather than an invented number.

This issue owns the cross-route response-time contract. It does not currently displace #610 or #661 from the execution queue.

Milestone

Current: M1 quality guard / measurement.

A response-time budget is required before the review route can be described as predictably usable, but this issue remains measurement-first until the first real-user and QA baselines identify where the time is spent.

Owning outcome

A user handing FOMO Kernel transaction history can understand how long it should take to receive the first useful review card, while maintainers can tell whether a slow run is caused by the product path or by the formal QA harness.

Current behavior / evidence

• Owner observation, 2026-07-31: producing one card currently feels very slow, and it is unclear whether the delay is real user experience or test-only ceremony.
• [P2·investigation·perf] Attribute the 5-10 min machine wait after the user answers (post-answer LLM turns + preview retry loop) #236 records an earlier real-review symptom: 5–10+ minutes of machine-only wait after the user finishes the questions, with preview rejection/rewrite loops and repeated context processing as leading hypotheses.
• [QA·dogfood] Judgment-quality dogfood — does a real consider answer now carry a supported insight? #610 records wall-clock for the card-free consider route and feeds that baseline here. It does not cover transaction history → full review card.
• The formal QA path adds version refresh, isolation checks, receipts, verification, archive, and privacy lint (docs/qa-runbook.md). Those are necessary maintainer costs but are not product latency and must not be mixed into the user-facing budget.

Decision — two lanes, two clocks

Lane A — real-user product latency

Measure the product lifecycle only. For a transaction-history review, record these timestamps live:

1. Input accepted → first computed value visible: the transaction source is accepted until the user sees the first engine-computed diagnosis, question, or other route-defined visible value.
2. Input accepted → private preview card delivered: the primary end-to-end product KPI for “交易记录到出卡.”
3. Last required answer → private preview card delivered: the critical machine-only wait already implicated by [P2·investigation·perf] Attribute the 5-10 min machine wait after the user answers (post-answer LLM turns + preview retry loop) #236.
4. Rule choice/skip submitted → final private card delivered: finalization latency.
5. Human response time: reported separately and excluded from machine latency; it remains part of total elapsed journey time.

The primary budget must not use finalize completion alone as “card delivered.” Delivery requires observable presentation evidence; a generated file is not enough.

Lane B — QA/test operator latency

A QA run records the same product segments, tagged by evidence lane:

• owner_live
• agent_with_owner_verdict
• agent_simulated

It also records a separate QA overhead clock for:

• latest-main refresh / version gate;
• isolated-root setup and isolate-check;
• optional contract preflight;
• receipt event writes;
• receipt verification;
• manifest/archive work;
• privacy lint and public finding preparation.

QA overhead is useful for improving maintainer throughput, but it is never included in the user-facing response-time budget. An agent_simulated run is a performance/debug data point, not proof of real-user UX.

Measurement record

Each measured run should preserve, without private trade content:

• tested main@SHA, client/host, route, and evidence lane;
• transaction-history vs snapshot input;
• coarse input-size bucket, never public transaction details;
• cold/warm dependency state and whether price/FX lookup ran or degraded;
• live segment durations above;
• number of agent/LLM turns after the last user answer;
• preview attempts, rejection count, and structured rejection reasons;
• external lookup/tool-call count;
• whether the first card was actually presented and by which delivery mode;
• QA overhead durations, when applicable, in a separate block.

Timings must be recorded as events happen. No backfilled timestamp burst (#278).

Performance review rule

A change touching an active route must state and verify expected latency impact when it adds or expands any of the following:

• network lookup;
• agent/LLM or tool round trip;
• required user question;
• validator retry surface;
• generated artifact or presentation step;
• per-position/per-row work that can grow with input size.

Review the route budget at each owner-live acceptance campaign and before a release claim. A material regression reopens this owner even when correctness tests remain green.

First measurement slice

1. Run one private owner_live transaction-history review on an explicitly frozen latest-main SHA and record the product timestamps live.
2. Run one agent_simulated or formal-QA transaction-history review on the same route/host class and record product time and QA overhead separately.
3. Compare the two traces to classify the dominant delay:
   • engine/network;
   • agent generation/context processing;
   • preview rejection/rewrite;
   • tool round trips;
   • QA-only ceremony.
4. Post only de-identified segment durations and failure shapes here.
5. Use the result to choose the next smallest fix; do not begin with broad instrumentation or a runtime timeout.

Interim guardrails already in force

• [design·M1] Bounded market-context lookup — verify why-now without inventing the user's motive #601's bounded market lookup: one primary lookup and at most one countercheck.
• SKILL.md rule 8: no multi-tool production for ordinary freeform answers.
• [design·M1] Decision-first TradeEvaluation answer — one supported judgment, not a disclosure dump #579/[perf·arch] 砍「答完→出卡」5–10 分鐘的主體：引擎生成 inferred-thesis 骨架 + prepare 內建契約知識（#236 主刀） #251 compression discipline: generation volume is a known latency source.
• [P2·investigation·perf] Attribute the 5-10 min machine wait after the user answers (post-answer LLM turns + preview retry loop) #236 remains the focused owner for post-answer preview/retry attribution; findings there roll up to this cross-route budget.

Non-goals

• No invented SLA number before measurement.
• No runtime timer that truncates an answer mid-delivery in the first cut.
• No weakening truth-critical gates to make the clock look better.
• No treating test-drive or automated preflight timing as real-user acceptance.
• No combining human think time, product machine time, and QA ceremony into one misleading total.

Acceptance criteria — first round

• One owner_live transaction-history review records all product segments live.
• One QA/test transaction-history review records the same product segments plus separate QA overhead.
• The issue contains a de-identified comparison identifying the dominant latency source(s).
• [P2·investigation·perf] Attribute the 5-10 min machine wait after the user answers (post-answer LLM turns + preview retry loop) #236 receives the post-last-answer attribution evidence if retries/generation remain material.
• Owner rules an initial per-route budget for at least: input accepted → preview delivered; last answer → preview delivered; rule choice → final card delivered.
• The approved budget lands in the owning route/reference documentation.
• A repeatable milestone/release regression review is documented without requiring private data.

Dependencies

• [P2·investigation·perf] Attribute the 5-10 min machine wait after the user answers (post-answer LLM turns + preview retry loop) #236 — focused post-answer machine-wait attribution.
• [QA·dogfood] Judgment-quality dogfood — does a real consider answer now carry a supported insight? #610 — first consider wall-clock baseline, not the review-card baseline.
• [QA·M1] Owner-live TradeEvaluation acceptance #488 — future formal owner-live acceptance campaign.
• [research·UX] 定义交易流程中的必要摩擦：何时必须追问、预览、承诺，何时应快速结束 #507 — owns which interaction steps are necessary; this issue owns how long the accepted steps may take.

Privacy boundary

Real trades, holdings, amounts, dates, motives, answers, and private repository content stay local. Public evidence may include only route, SHA, host/evidence lane, coarse input-size classification, segment durations, retry/tool counts, delivery mode, and de-identified failure shape.

[atomchung]

2026-07-31 owner delta

Expanded this owner from a generic per-route timing budget into an explicit real-user vs QA/test split after the owner reported that a transaction-history review currently feels very slow and it is impossible to tell whether the delay belongs to the product or to the test harness.

Changes now in the issue body:

• primary review KPI is input accepted → private preview card delivered;
• last required answer → preview delivered is tracked separately as machine-only wait;
• human response time and rule choice → final card are separate segments;
• formal QA setup/receipt/verification/archive/privacy work has its own overhead clock and is excluded from user latency;
• owner_live, agent_with_owner_verdict, and agent_simulated evidence are not interchangeable;
• route-shaping changes that add lookups, tool/LLM turns, questions, retries, artifacts, or input-scaled work must assess latency impact;
• the first slice is one owner-live transaction review plus one QA/test transaction review, followed by an owner-set initial budget rather than an invented SLA.

This reuses #603 rather than opening a duplicate. #236 remains the focused investigation for the known 5–10+ minute post-answer wait; #610 only supplies the separate consider baseline.

[atomchung]

Current determination — QA can inflate the session, but it does not explain the core post-answer wait

Repository evidence now supports a partial answer before the matched baseline in this issue is run.

What can already be separated

tools/ux_receipt.py timestamps every presentation row and has an explicit answers_received marker written when the final required answer arrives, followed by card_presented(stage=preview). Therefore an eligible trace can directly measure last required answer → preview card actually presented. This segment is inside the product path; version refresh, isolation, preflight, archive, manifest and privacy-lint work do not belong in it.

The current receipt does not have an input_accepted marker or explicit QA-phase markers. Formal QA setup/closure also happens outside the receipt. Therefore existing traces cannot by themselves derive the complete transaction input → card product duration or the total QA overhead. A whole-conversation wall clock remains ambiguous.

Existing evidence about the dominant delay

#236 already measured two interactive real-trade dogfood runs rather than test-drive fixtures:

• one steady-state run took 7m44s from final answer to card, with one successful preview, zero rejects, and preview/finalize engine execution around 0.16–0.21s;
• an earlier run took 19m25s, also with zero preview rejects;
• the clean 7m44s trace attributed roughly 60% to agent narrative/thesis generation, 15–20% to payload reprocessing/round-trip latency, around 12% to runtime contract/source investigation, and around 7% to rendering.

This means the engine is not the main source and QA ceremony is not sufficient to explain the user-visible wait. The principal observed cost is the product's agent orchestration/generation path. QA can still add substantial time around that path and can contaminate a sample when the maintainer stops to inspect code or repair fixtures.

A later fomo-qa run recorded about 3m22s from answers_received to preview presentation, but explicitly mixed maintainer schema debugging into the interval; it is evidence that reject/rewrite can happen, not a clean product-latency baseline.

Classification rule for the next comparison

For each run, classify elapsed time into four non-overlapping clocks:

1. QA setup/closure — latest-main refresh, isolation, preflight, receipt/manifest/archive/privacy work required only to produce acceptance evidence.
2. Product machine time — normalization, engine/network work, agent generation/context processing, validator/retry, render and actual delivery required for a normal user to receive the card.
3. Human response time — reading and answering questions or choosing a rule.
4. Investigation/debug time — source inspection, fixture/schema repair, reruns caused by the maintainer changing the test itself. This invalidates a clean latency sample and must not be silently assigned to either product or QA overhead.

Until input_accepted/QA phase markers exist, use the existing receipt for answer→preview and presentation evidence, plus contemporaneous shell/stopwatch timestamps for input acceptance and QA setup/closure. Do not add broad canonical instrumentation before the first matched pair proves it is needed.

Present conclusion

• Yes: formal QA can make the whole session look much slower than the real product, especially before input and after final delivery.
• No: current evidence does not support treating most of the critical post-answer wait as QA-only. A real user can experience the same multi-minute agent generation/context cost.
• The next smallest proof remains the matched pair already specified in this issue: one clean private user-path run and one formal-QA run on the same SHA, route, host class and coarse input-size bucket.