[fix] DB connection resilience: pool_pre_ping so idle connections don't stall the ETL - #14
Closed
chondl wants to merge 1 commit into
Closed
[fix] DB connection resilience: pool_pre_ping so idle connections don't stall the ETL#14chondl wants to merge 1 commit into
chondl wants to merge 1 commit into
Conversation
A pooled connection left idle across the hourly-cron gap goes stale (Cloud SQL / db-f1-micro / the Cloud SQL proxy reap idle connections). The next query then raises 'server closed the connection unexpectedly', which 500s the ETL trigger (/v3/site/update_curr_year) and silently stalls ingestion — on the mirror this stopped offseason match schedules from appearing. pool_pre_ping reconnects transparently; pool_recycle=1800 drops old connections proactively. Deployed and verified on the staging mirror (backend rev 00014): the update endpoint returns 200 repeatedly where it had been 500ing, and 2026iri ingested its full match schedule.
Owner
Author
|
Consolidated: this pool_pre_ping fix was folded into #25 [06] ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A pooled SQLAlchemy connection left idle across the hourly-cron gap goes stale (Cloud SQL / db-f1-micro / the Cloud SQL proxy reap idle connections). The next query raises
psycopg2.OperationalError: server closed the connection unexpectedly, which 500s/v3/site/update_curr_year(the scheduler was returning error code 13) and silently stalls all ingestion.On the mirror this is why an offseason event's match schedule (2026 IRI: 75 matches, 15 played) stopped appearing — the ETL cycle kept crashing on the stale connection before it could ingest.
Fix
create_engine(CONN_STR, pool_pre_ping=True, pool_recycle=1800). pre_ping issues a lightweight liveness check and transparently reconnects a dead pooled connection; recycle proactively drops connections older than 30 min. Not offseason-specific — a general infra fix for any Postgres/CockroachDB deploy.Verification
Deployed to the staging mirror (backend rev 00014).
/v3/site/update_curr_yearreturns 200 repeatedly where it had been 500ing; 2026iri ingested its full 75-match schedule with predictions + results.Notes
stagingbranch (deployed) as a direct commit; this PR captures the same fix againstmasterfor review + easy promotion to main.