[defect·P2] A permanently-unpriced symbol makes the cache refuse the exact request that produced it — every same-day command re-fetches the whole universe

[atomchung]

Status

Discussion — mechanism-level finding from PR #684's investigation into #665. No real trade data anywhere below; no maintainer decision yet.

Owning outcome

A same-day prepare bundle should answer a later same-day resume/consider/add-cash on the same book for free — that reuse is the entire point of the same-day market-data cache (#605 §D). It should hold for every book, not only ones where every requested symbol eventually prices.

Mechanism

skills/fomo-kernel/engine/market_data.py (line numbers against origin/main@e662164):

• MarketDataBundle.covers (:322-366) requires the bundle's priced set to be a superset of the request's instruments/benchmarks (:353-354: if set(request["instruments"]) - priced: return False). priced (:288-295) is the set of columns with at least one non-NaN close. A permanently-unpriced symbol — delisted, or simply misspelled, both ordinary in a real TW/US book — has an all-NaN column and is therefore never in priced, for any bundle fetched at any time.
• The superset rule is deliberate and documented at :345-350: "a bundle that requested {GOOD, DEAD} and priced only GOOD would otherwise 'cover' a later request for DEAD, hand back a bundle with no DEAD price, and suppress the retry that a transient outage deserves (external review, finding 5)." That rationale is correct for a transient outage — retrying is the whole point.
• _cache_load (:763-778), the same-day disk cache, gates every lookup on bundle.covers(request) (:776). The sharp edge: once a permanently-unpriced symbol is in the request, covers returns False even against the exact request that produced the cached bundle — a request for {GOOD, DEAD} stores a bundle whose own priced set excludes DEAD, so re-asking that identical request the same day still fails covers and misses the cache it just wrote.
• _from_yahoo (:630-679) falls through to a fresh fetch of request_universe(request) — the whole universe, not just the missing symbol (:650-652) — whenever _cache_load returns None (:646-648).
• resolve (:795-844) routes both the in-process memo (:837-840) and the disk cache through the same covers call, so neither reuse path is available once a permanently-unpriced symbol is anywhere in the request.

Net effect: a book with one delisted or misspelled symbol anywhere in its instruments loses same-day caching entirely, for itself. Every prepare, resume, consider, and add-cash call on that book re-fetches every other symbol too, at full provider cost, for no informational gain — the permanently-unpriced symbol was never going to price differently on a second same-day call.

Why this is not simply #665

#665 (closed, fixed by PR #684) was one symptom: add-cash's drift gate mistook the consequence of this re-fetch — a second live resolution disagreeing with the first because the provider moved between calls — for the anchor itself having broken something. PR #684's fix is a frozen-frame carve-out scoped to add-cash only (amending_session=True re-entering _prepare_session, _from_frozen_frame in market_data.py): it makes add-cash reuse the exact frozen frame regardless of what covers says, but does nothing for resume, consider, or a second plain prepare the same day on the same book. The re-fetch-the-whole-universe cost this issue describes is still paid by every one of those routes.

Evidence

Read against origin/main@e662164 (post-#684/#685 merge). No real trade data; this is a code-mechanism finding traced by direct reading of covers/_cache_load/_from_yahoo/resolve, not a live-provider reproduction.

Cross-references

• Found during PR fix(add-cash): the drift gate compares frozen source facts, so the card-beat cash answer is recordable (closes #665) #684's investigation of [defect·P1] add-cash refuses the session it exists for: the drift gate compares question_queue and state_snapshot whole, so the card-beat cash answer is unrecordable #665 (this repository's own fix(add-cash): the drift gate compares frozen source facts, so the card-beat cash answer is recordable (closes #665) #684 PR description states the general cost explicitly as the reason [defect·P1] add-cash refuses the session it exists for: the drift gate compares question_queue and state_snapshot whole, so the card-beat cash answer is unrecordable #665's fix is scoped to add-cash alone).
• [architecture·implementation] One market-data resolver — minimum Yahoo calls, one fact bundle for prepare and consider #605 (closed) — the architecture issue whose deliberate design (covers's superset rule, "external review, finding 5") is the direct cause of this side effect. Not a flaw in [architecture·implementation] One market-data resolver — minimum Yahoo calls, one fact bundle for prepare and consider #605's own decision — a gap in what it distinguishes between a transiently- and a permanently-unpriced symbol.
• [bug·P2] --root bounds session state but not the market-data cache — the prepare route resolves its own root #627 (closed) — a different, already-fixed defect in the same same-day cache (root scoping bypassed session isolation); unrelated mechanism, same file and same cache.
• [defect·P1] add-cash refuses the session it exists for: the drift gate compares question_queue and state_snapshot whole, so the card-beat cash answer is unrecordable #665 (closed) / PR fix(add-cash): the drift gate compares frozen source facts, so the card-beat cash answer is recordable (closes #665) #684 (merged) — where this was found, as a narrower symptom already fixed for add-cash alone via the frozen-frame carve-out.