[blob-store 3/5] Pipeline state as a snapshot blob; DB becomes a downstream consumer

[chondl]

Track 2 · blob-store stack — PR 3 of 5. Base: blob-gc (#7); merge after #7. Stack order: #2 → #7 → #9 → #10 → #11. This is the resilience payoff: the pipeline survives a DB outage. No cross-track conflicts.

---

Summary

The website already serves from bucket blobs (content-addressed blobs behind a manifest, from the stacked bucket-first-serving and blob-gc PRs). This PR takes the next step: it makes the update/publish pipeline itself resilient to database unavailability, and demotes the relational DB from a hard dependency of the pipeline to a downstream consumer that serves only the public /v3 API.

Today the DB is used as the pipeline's serialization format, not a query engine: read_objs() loads the whole current year from the DB at the start of every cycle, and write_objs() diff-upserts it back at the end. If the DB is unavailable — the failure mode behind the June production outage — the entire fetch/compute/publish cycle fails and the site goes stale even though it serves from blobs.

This PR replaces that seam:

1. Snapshot blob as the source of truth. At the end of each successful current-year cycle the full in-memory state (the objs tuple — year, team_years, events, team_events, matches, etags — plus teams) is persisted as one compressed object at state/snapshot.<year>, written before the blob publish.
2. Cycle start loads from the snapshot. If no snapshot exists (first deploy / migration), the pipeline falls back to the DB read path; state converges after one cycle.
3. DB writes move off the hot path. The diffed DB upsert still runs — the public /v3 API still serves from the DB — but only after the snapshot and blob publish, and it is now non-fatal: a DB outage is logged and the cycle continues.

Per-change rationale

src/google/snapshot.py (new) — deterministic state serialization

Serializes the objs tuple + teams to json + zlib, consistent with the existing blob conventions (compress() in storage.py). Chosen because it is human-inspectable, dependency-free, and already the project's blob format.

• Deterministic: every collection is sorted by primary key before serialization, so the same state always produces the same bytes (verified by re-serializing a round-tripped snapshot).
• Versioned + validated on read: the payload carries a top-level "schema" field (currently 1). deserialize() checks it and read_snapshot() falls back to the DB path (logging why) on a mismatch, missing objs, or any corrupt/short payload — it does not silently None-fill missing columns via from_dict. Bump SNAPSHOT_SCHEMA on any incompatible ORM/layout change (e.g. an added column) so a pre-deploy snapshot is rejected rather than read with the new column silently null forever.
• Type fidelity: enum-typed columns (comp_level, status, winner, type, ...) are detected generically from the SQLAlchemy ORM and coerced back to their Enum members on load, so a snapshot-loaded object is byte-for-byte indistinguishable from a DB-loaded one. This matters because the honest-diff gate compares str(obj).
• Atomicity uses the tmp-then-copy pattern (tmp key -> server-side copy), mirroring the manifest-last publish. The tmp key carries a per-writer pid+uuid suffix so two concurrent publishers (double scheduler fire / overlapping revisions) cannot race on a shared staging blob.

src/google/storage.py — publish from memory, tolerate DB loss

write_objs() now renders the core current-year blobs (team_years, events, event/{key}, team_to_events, ...) purely from the in-memory objs + teams passed in, so the publish no longer needs the DB. The remaining cross-year enrichment reads (teams/all, events/all, per-team pages, noteworthy/upcoming matches) are wrapped best-effort: if the DB is down they are skipped, and because publishing is content-addressed behind the manifest, the previous good version of each skipped blob is simply carried forward — no stale-overwrite, no cycle failure.

Two consistency refinements carried from the bucket-first event-content gate: (1) the gate needs the pre-cycle state to tell which events changed, so process_year now deepcopys the cycle-start objs and passes them as orig (rather than None); without this, every event blob would re-upload each cycle as the embedded year stats drift (measured 215 → 0 on the rig). The deepcopy adds ~2.8 s to a partial cycle — cheaper than re-uploading 215 blobs and re-warming the edge cache. (2) each team's current-year row on its team/{num} page is now taken from the in-memory objs (fresh this cycle) rather than the persisted read (previous cycle), so a team page's current EPA no longer lags the team_years list by one cycle.

src/data/main.py + src/data/utils.py — wire the new seam

• update_curr_year() loads state from the snapshot and only falls back to get_teams_db() + read_objs() when no snapshot is present.
• process_year() writes the snapshot, then publishes blobs, then does the DB upsert inside a non-fatal try.
• Healing is preserved. The honest-diff for the DB write is computed against a fresh, best-effort DB read (not the snapshot), so once the DB returns, the next cycle upserts exactly the rows the DB is missing. Diffing against the snapshot would silently skip a DB that fell behind during an outage. This read is off the critical path (publish has already happened) and is skipped entirely when the DB is down.
• read_objs() now returns its dicts in primary-key order to match the snapshot's deterministic ordering, so the DB and snapshot load paths publish byte-identical output (and publishing becomes deterministic regardless of DB row order).

Deferred (follow-up)

Cross-year seed reads used for EPA initialization (prior years' team_years, norm_epa aggregates) still read the DB, wrapped best-effort. They are static reference data and belong in hist/ blobs; moving them there is a separate, smaller change and keeps this diff focused.

Verification (local rig: CockroachDB + fake-gcs, full 2026 season)

Byte-identical replay. A full cycle loaded from the DB vs. from the snapshot produced identical published content across all 3950 logical blobs (compared via manifest content-hashes): 0 mismatches.

Headline test — DB down. With the CockroachDB container stopped, one update cycle:

Read Snapshot       0:00:01.47      <- loaded state, no DB
2026 Write Snapshot 0:00:08.83
2026 Write Storage  0:00:03.67      <- blobs published from memory
2026 Write DB       0:00:00.002     <- failed fast, logged, skipped
CYCLE_RETURNED_OK in 18.42s
MANIFEST_ADVANCED: True
SNAPSHOT_ADVANCED: True
psycopg2.OperationalError: connection ... port 26257 failed: Connection refused

The cycle fetched TBA, computed EPA, and published blobs + snapshot successfully; the only failure was the logged, non-fatal DB write.

Healing. Deleted 25 team_years rows from the DB; the next DB-up cycle's diff (computed against the live DB) restored the count from 3699 back to 3724.

Cold start. With no snapshot present the pipeline falls back to get_teams_db() + read_objs(), runs a full cycle, and writes the first snapshot; the following cycle loads from it.

Shared smoke suite: 10/10.

Timings (this machine, static 2026 data)

Step	DB path	Snapshot path
Load state (Load Teams + Read Objs)	~2.7 s	Read Snapshot ~1.4 s
Full cycle	~25.0 s	~23.4 s

Loading state from the snapshot roughly halves the state-load step and removes it from the resilience-critical path. The snapshot write adds ~8.8 s per cycle (serialize + upload of the full ~23 MB state); a delta/streamed snapshot is a reasonable future optimization but is intentionally out of scope here. The cycle-start deepcopy for the event-content gate adds ~2.8 s. The best-effort DB read for the heal-diff is the residual DB touch on a healthy cycle and is skipped during an outage.

Notes

• No behavior change to the public /v3 API or to the frontend.
• No tests are added on this branch, consistent with the stack's convention.

[chondl]

Superseded by #21 — reopened as #21 [03] on cph-state-snapshot.