[test-infra] The #357 cash-anchor test block runs price-degraded by default — a moving-provider harness now exists but covers only one of its tests

[atomchung]

Status

Discussion — process/test-coverage finding from PR #684's investigation into #665. No real trade data anywhere below; no maintainer decision yet.

Owning outcome

A test that exercises an engine-priced code path — one that calls market_data.resolve with network access notionally allowed — runs against a harness that can actually disagree with itself between two calls, at least once per behavior it claims to cover, so a defect that only a second live resolution can expose is not structurally invisible to the offline suite.

Mechanism

tests/test_review_v2.py (line numbers against origin/main@e662164):

• The section banner at :1920, # --- The cash anchor is a stated gap, and answerable after the card (#357) ----, opens the pre-existing cash-anchor/add-cash test block. Every test in it before PR fix(add-cash): the drift gate compares frozen source facts, so the card-beat cash answer is recordable (closes #665) #684 — and all but one after — builds its environment with _offline_env (:1936-1946), which injects an ImportError yfinance stub so market_data.resolve never reaches a provider at all (network_allowed short-circuits before any fetch).
• With nothing retrieved, there is no second retrieval to disagree with the first. [defect·P1] add-cash refuses the session it exists for: the drift gate compares question_queue and state_snapshot whole, so the card-beat cash answer is unrecordable #665's actual cause — add-cash re-entering the engine and getting a second, different live resolution because the provider moved between calls — cannot occur under _offline_env by construction: a stub that always raises ImportError returns the same degraded answer every time, so a drift-comparison test built on it exercises the comparison logic on identical inputs and can only prove the gate behaves when there is nothing to disagree about.
• PR fix(add-cash): the drift gate compares frozen source facts, so the card-beat cash answer is recordable (closes #665) #684 added _priced_env (:2148-, using a usercustomize-injected moving-provider module, following test_consider.py's existing pattern for a live-like harness) and gave exactly one test in this block — test_the_card_beat_cash_answer_is_recordable_on_an_engine_priced_review (:2190) — a priced lane. Every other test in the #357 block, before and after PR fix(add-cash): the drift gate compares frozen source facts, so the card-beat cash answer is recordable (closes #665) #684, still uses _offline_env (for example :2071, :2358, :2411, :2520, :2542).

This is offered as the general finding, not as a defect in the remaining _offline_env tests individually: several of those are testing a refusal shape (a grown transaction file, a gone frame, a replay) that does not need a moving provider to exercise, and converting them would add nothing. The narrower, more durable point is that the harness gap itself was invisible from inside the suite until #665 was reproduced by running the real CLI against real Yahoo access — nothing in the offline suite could have shown the same symptom, no matter how many _offline_env tests were added to the block. Any other engine-priced behavior elsewhere in this repository's tests that is asserted only under an _offline_env-equivalent stub carries the same structural blind spot #665 exposed, until it is specifically checked against a moving-provider harness.

Evidence

Read against origin/main@e662164; confirmed by grep (_offline_env, _priced_env, 357, usercustomize) across tests/test_review_v2.py and by reading both harness functions and the one test PR #684 converted. No real trade data — this is a test-infrastructure finding about which harness a given test uses, not a data reproduction.

Cross-references

• [defect·P1] add-cash refuses the session it exists for: the drift gate compares question_queue and state_snapshot whole, so the card-beat cash answer is unrecordable #665 (closed) / PR fix(add-cash): the drift gate compares frozen source facts, so the card-beat cash answer is recordable (closes #665) #684 (merged) — the concrete defect this blind spot let survive a green suite; PR fix(add-cash): the drift gate compares frozen source facts, so the card-beat cash answer is recordable (closes #665) #684's own description states the causal chain this issue generalizes from.
• [test-infra] test_consider.py is red standalone on main — nine tests depend on run_all.py assigning TR_OFFLINE #620 (closed) — the closest existing precedent for this class of finding ("test_consider.py is red standalone on main — nine tests depend on run_all.py assigning TR_OFFLINE"): a different symptom (suite-ordering dependency) of the same underlying fact that this repository's tests lean heavily on an offline/stubbed posture for determinism, which trades one kind of flakiness for this kind of blind spot.
• [chore·tests·P3] test_review_v2.py is 8,361 lines and ~20 unrelated concerns sharing one file by accretion #464 (open) — tests/test_review_v2.py's size and lack of internal structure is tracked there as a separate, deliberately deferred concern (splitting it now maximizes merge contention across parallel sessions). This issue is about which harness a test in that file uses, not the file's size or organization, and does not depend on [chore·tests·P3] test_review_v2.py is 8,361 lines and ~20 unrelated concerns sharing one file by accretion #464 being actioned first or block it.
• [design·process] Only splits stops a new workflow shipping without handling a basic fact — FX, pricing basis and classification have no equivalent #607 (open) — a structurally similar process gap one layer up: no declared matrix of "basic fact × route" coverage for a new review.py subcommand, so a new workflow can ship having silently skipped FX/pricing-basis/classification handling the way it cannot for splits. This issue is the same shape applied to existing tests' harness choice rather than to route coverage, and is not a duplicate of it.