[06] Postgres compatibility (dialect, BigInteger timestamps, DATABASE_URL, pool_pre_ping) - #25
Open
chondl wants to merge 4 commits into
Open
[06] Postgres compatibility (dialect, BigInteger timestamps, DATABASE_URL, pool_pre_ping)#25chondl wants to merge 4 commits into
chondl wants to merge 4 commits into
Conversation
Lets the backend run against plain PostgreSQL (e.g. Cloud SQL) instead of CockroachDB by exporting DATABASE_URL, without disturbing the existing PROD CockroachDB path or local dev default. Retry behavior is selected from the engine dialect (see src/db/transaction.py).
Introduce src/db/transaction.py with a run_transaction wrapper that keeps the CockroachDB behavior unchanged (delegates to sqlalchemy_cockroachdb, imported lazily) while adding a plain-SQLAlchemy transaction path for other dialects (PostgreSQL). The Postgres path preserves retry-on-serialization-failure (SQLSTATE 40001) semantics. Repoint the 13 db read/write/function modules to import run_transaction from src.db.transaction instead of sqlalchemy_cockroachdb.
On Postgres, SQLAlchemy Integer maps to int4 (32-bit). TBA returns a ~1900 placeholder Unix timestamp (-2208988800) for matches/events with an unknown time, which underflows int32 and aborts historical builds at 2006 with psycopg2.errors.NumericValueOutOfRange. CockroachDB INT is 64-bit, so this never surfaced on the production database. Widen the four timestamp columns (match.time, match.predicted_time, event.time, team_event.time) to BigInteger; this matches CockroachDB's width and is future-proof past the 2038 int32 limit.
A pooled connection left idle across the hourly-cron gap goes stale (Cloud SQL / db-f1-micro / the Cloud SQL proxy reap idle connections). The next query then raises 'server closed the connection unexpectedly', which 500s the ETL trigger (/v3/site/update_curr_year) and silently stalls ingestion — on the mirror this stopped offseason match schedules from appearing. pool_pre_ping reconnects transparently; pool_recycle=1800 drops old connections proactively. Deployed and verified on the staging mirror (backend rev 00014): the update endpoint returns 200 repeatedly where it had been 500ing, and 2026iri ingested its full match schedule.
chondl
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Distinct re-arch step: Postgres/Cloud SQL support. Part of the cph-staging series.