[QA·dogfood] Judgment-quality dogfood — does a real consider answer now carry a supported insight?

[atomchung]

Status

Ready — owner walk with agent support. The first live test of the judgment-quality layer that landed 2026-07-30.

#488's ruling deferred the next formal acceptance campaign "until the agent side has enough judgment quality to be worth re-walking." This issue is that test, deliberately lighter than a formal campaign: one real decision, scored honestly. It does not replace #488's future exact-SHA re-walk and must not be cited as it.

What changed since the last dogfood

The 2026-07-30 dogfood verdict on a live consider answer was: for-side stated the obvious, against-side was five boilerplate lines, 「整體來看，回答毫無意義」. Since then, on main@94b7e5f:

• concentration numbers were fixed ([bug·data] consider's driver/sector classification table has zero foreign-listing coverage — same company, different ticker string, falls through as 未分類 #598/[bug·data] consider's legacy CSV lane silently mixes currencies (missing FX defaults to 1.0) — a live-fetch path already exists but isn't wired to it #600, PR fix(consider): the concentration numbers state what they could not read, and refuse a book they cannot add up (closes #598) (closes #600) (refs #599) #602/refactor(consider): a state carries the positions its own concentration readings could not read (refs #598) (refs #599) #606) — the groundedness bugs behind "the numbers look wrong";
• references/market-lookup.md (PR docs(contract): market context reaches a consider answer only through the bounded lookup contract (closes #601) #604, [design·M1] Bounded market-context lookup — verify why-now without inventing the user's motive #601): L0 standing position context, L1 bounded event lookup with motive confirmation, L2 dimension lookup, sufficiency-floor stop discipline;
• references/trade-consequence.md decision-first section (PR docs(contract): a consider answer leads with one supported decision tension, not a disclosure dump (closes #579) #608, [design·M1] Decision-first TradeEvaluation answer — one supported judgment, not a disclosure dump #579): salience order, two-paragraph shape, compression, the rebuttal requirement, and a same-payload good/bad contrast.

This walk answers: does a real answer now clear the bar those changes were built for?

The walk

1. One real decision. The owner brings one trade they are genuinely considering (or most recently considered). The agent runs the full current flow: consider with --decision-context, L0 position packet, L1 lookup if triggered (with the motive-confirmation question), decision-first answer. Receipted via the card-free consider route where the environment allows (qa discipline per docs/qa-runbook.md if run as a formal QA campaign; a private real-book walk records the generic verdict only).
2. Score the answer — the four checks (from the [design·M1] Decision-first TradeEvaluation answer — one supported judgment, not a disclosure dump #579 ruling; each pass/fail plus one sentence why):
   • Cover the limitation sentences — does the answer still stand on its own?
   • Does it say at least one thing the owner did not already know, or had not connected?
   • Does the counter-case directly engage the lead's strongest support (a real rebuttal, not a relabeled disclaimer)?
   • Is the owner's next step more concrete than before reading?
3. Record live wall-clock segments ([design·perf] Response-time budget by route — separate real-user latency from QA/test overhead #603's first baseline — no backfill, per [QA·process] ux receipt trace was backfilled in one burst: all 10 events stamped within 1s, defeating latency instrumentation #278): the consider call, each lookup, total time to the delivered answer.
4. Walk at least two market-lookup scenes from tests/agent/market-lookup-scenes.md as they naturally occur (scene 1 or 6 for the L1 path, scene 4 or 7 for the L0 boundary), and record pass/fail against the scene's own criteria.

What the results feed

• Four-check scores → this issue and [design·M1] Decision-first TradeEvaluation answer — one supported judgment, not a disclosure dump #579's evidence loop. A failure is not a bad outcome: converting its exact structure into the next synthetic episode is the sequencing [design·M1] Decision-first TradeEvaluation answer — one supported judgment, not a disclosure dump #579's own ruling asks for.
• Wall-clock baseline → [design·perf] Response-time budget by route — separate real-user latency from QA/test overhead #603.
• Lookup behavior → [design·M1] Bounded market-context lookup — verify why-now without inventing the user's motive #601's acceptance line "one decision where the agent either performed a justified lookup or correctly declined."
• Real context-bearing usage → the [research·roadmap] Staged direction: pre-trade challenge → decision memory → rule learning #475 Phase 1 usage gate. If the owner judges the interaction genuinely used and revisited, that unblocks [design·M2·parked] consider surfaces the same ticker's prior evaluations — the first consumer of decision memory #609 (prior-evaluations supply, currently parked).
• A pass on all four checks → evidence that a formal [QA·M1] Owner-live TradeEvaluation acceptance #488 re-walk is now worth scheduling.

Success condition

Not four green checks. Success = an honest per-check verdict on a real decision, timings recorded live, and every failure converted into a named next cut. A dishonest pass would poison #579's evidence-first loop.

Privacy

The real trade, book, motive, and answer stay local. Only the generic pass/fail structure, timings, and de-identified failure shapes are posted here (tools/privacy_lint.py before posting, per the QA runbook).

Refs #579, #601, #603, #488 (deferred formal acceptance), #475 (Phase 1 usage gate), #609 (unlocks on that gate).

[atomchung]

M1 cleanup audit routing, 2026-07-30 (main@dabf709): run this walk after #612 and #618, not before.

#611 and #616 have moved the runtime since the candidate SHA named in this body. The next two bounded M1 leaves are now:

1. [defect·P1] prepare aggregates a missing FX rate as 1.0 — #602's refusal never reached the review lane #612 — fail closed before prepare can aggregate a missing held-currency FX rate as 1.0;
2. [implementation·M1] A consider answer states which day's prices it was computed from #618 — freeze the price observation date and make the consider answer state it.

After both land, freeze one new exact main SHA and run this issue unchanged in purpose: one real decision, four judgment checks, timings and lookup scenes. The private facts stay local. This sequencing is now the active route in #27.

This is not a new prerequisite chain for other work and does not reactivate formal #488 acceptance. It only avoids spending the owner walk on an answer whose arithmetic or price-date attribution is already known to be incomplete.

[atomchung]

Routing satisfied — candidate SHA frozen at main@3e174d9

The comment above parked this walk behind #612 and #618. Both have landed, and #629 landed between them, which changes what this walk is actually testing. Everything below is agent-side setup; the walk itself needs the owner and cannot be delegated.

What moved since this body was written

Issue	Landed	What it changes for this walk
#612	2f5b0ec (PR #640)	A mixed-currency book with a missing held-currency rate now refuses instead of producing a card. On the shipped tw_mixed persona the old path read its largest position as 87.8% of a book that is really 57.1%, and the ranking beneath it was noise — MSFT at 0.5% where it is really 11.1%. If the walk uses a mixed book, this is the arithmetic that used to be wrong under it.
#629	6723db1 (PR #641)	consider with no retrievable prices now emits a recovery kit naming the instruments to look up, and refuses rather than answering on cost once recovery is declared dead. Measured on the momentum fixture: NVDA reads 59.8% on cost vs 46.0% on market, and the second and third positions invert.
#618	3e174d9 (PR #643)	The answer now carries basis.price_observations — per ticker plus a frame summary — and must_state makes the answer owe it. An unpriced run carries no date at all.

The practical effect on check 1 ("cover the limitation sentences — does the answer still stand on its own?"): the answer now owes one more fact than it did when this body was written. If the price day reads as noise rather than as something you can act on, that is a finding worth recording, not a nuisance.

The walk, unchanged in purpose

1. One real decision you are genuinely considering. Full current flow: consider with --decision-context, L0 position packet, L1 lookup if triggered (with the motive-confirmation question), decision-first answer.
2. The four checks, each pass/fail plus one sentence why:
   • Cover the limitation sentences — does the answer still stand on its own?
   • Does it say at least one thing you did not already know, or had not connected?
   • Does the counter-case directly engage the lead's strongest support — a real rebuttal, not a relabeled disclaimer?
   • Is your next step more concrete than before reading?
3. Wall-clock segments recorded live ([design·perf] Response-time budget by route — separate real-user latency from QA/test overhead #603's first baseline, no backfill): the consider call, each lookup, total to delivered answer.
4. At least two market-lookup scenes from tests/agent/market-lookup-scenes.md as they naturally occur — scene 1 or 6 for the L1 path, scene 4 or 7 for the L0 boundary — pass/fail against each scene's own criteria.

One thing to watch that is new: if the host cannot reach a price source, #629 now expects the agent to recover the closes before answering. Whether that actually happens is the instruction-layer half nothing offline can gate, and this walk is the first real observation of it. If the agent skips the recovery and relays a cost-basis answer anyway, that is the single most valuable finding this walk can produce — record it rather than re-running.

Success condition, restated

Not four green checks. Success is an honest per-check verdict on a real decision, timings recorded live, and every failure converted into a named next cut. A dishonest pass poisons #579's evidence-first loop.

Privacy

The real trade, book, motive and answer stay local. Only the generic pass/fail structure, timings, and de-identified failure shapes are posted here — tools/privacy_lint.py before posting, per docs/qa-runbook.md.

Two known gaps that could shape the walk, filed rather than fixed

• [defect·P2] #612's refusal names a repair the price-feed schema will not accept: there is no fx-only envelope #642 — [defect·P1] prepare aggregates a missing FX rate as 1.0 — #602's refusal never reached the review lane #612's refusal points at --prices' fx block as the repair, but the schema requires prices with minItems: 1, so there is no fx-only envelope. If your book is mixed-currency and the FX source is down, the named repair does not currently work.
• fix(card): the standing-rule line names this period's prescription, not the rule the user actually committed to #645 — a returning user whose prior commitment differs from this period's prescription is still told "the standing rule remains" and then shown a rule they never chose. Needs an owner decision between two readings before it can be implemented; it is on the review-card lane, not consider, so it should not affect this walk.

[atomchung]

Candidate SHA updated: main@9d56a2d (was 3e174d9). Post-merge CI green on 3.11 and 3.12.

Two more landed since the comment above, both relevant to this walk:

• [defect·P2] #612's refusal names a repair the price-feed schema will not accept: there is no fx-only envelope #642 (878c333) — [defect·P1] prepare aggregates a missing FX rate as 1.0 — #602's refusal never reached the review lane #612's refusal named a repair the schema would not accept: prices was required, so an fx-only envelope was impossible and a host that could look up a rate but not closes had no way out. prices is now optional. If your book is mixed-currency and the FX source is down during the walk, the repair the error names now works — verified end to end.
• [bug·data·P1] An already-adjusted broker export gets the split applied twice — 10× the real share count, and nothing detects it #582 (9d56a2d) — a transaction file whose own pre-split prices contradict its split basis is now asked about rather than reviewed. Only fires on tickers with a split in the window, only on pre-split rows, and only when every examined row agrees. If it fires during the walk on a real file, that is itself a finding worth recording; it names the ticker, the split, the factor, and the rows it measured.

Nothing else about this issue changes. The walk is unchanged in purpose: one real decision, the four judgment checks scored honestly, timings recorded live, two market-lookup scenes. Private facts stay local; only the generic verdict structure, timings and de-identified failure shapes get posted.

One addition to what is worth watching, from the design discussion recorded in #650: when the candidate rules are offered, note whether the engine's top-ranked candidate is the one you would actually pick. Nobody has measured that, and it is the single fact that decides whether the pre-commitment preview card earns its second full render — currently 5 differing lines out of 53–63 against the finalized card, all of them the rule sentence.