Skip to content

fix(cron): gate startup on schema readiness before opening the pooled context#329

Merged
hhvrc merged 1 commit into
developfrom
fix/cron-schema-ready-gate
Jul 1, 2026
Merged

fix(cron): gate startup on schema readiness before opening the pooled context#329
hhvrc merged 1 commit into
developfrom
fix/cron-schema-ready-gate

Conversation

@hhvrc

@hhvrc hhvrc commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Problem

The Cron host was failing every EmailOutboxDeliveryJob claim query in production with:

The data type name 'email_status' could not be found in the types that were loaded by Npgsql.

Root cause

OpenShockContext binds the Postgres enum types (email_status, email_type, …) to CLR enums by name via MapEnum<T>(). Npgsql resolves those names against the type catalog it loads on the pooled data source's first connection, then caches it for the life of the process.

The Cron host does not own migrations — the API is the sole migrator (API/Program.cs calls ApplyPendingOpenShockMigrations; Cron never did). So during the outbox rollout, Cron's data source loaded its type catalog before the API had applied the migration that creates those enums. The missing type was cached as absent, and every query using it failed permanently until the process was restarted.

This is a deploy-time race that recurs whenever a new enum consumed by Cron is added.

Fix

  • Add WaitForOpenShockSchemaReady(DatabaseOptions) in Common/OpenShockMiddlewareHelper.cs — polls GetPendingMigrationsAsync() with capped backoff and a timeout.
  • Call it in Cron/Program.cs right after UseCommonOpenShockMiddleware(), before Hangfire or any job opens the pooled context.

Cron performs no schema writes; the API stays the sole migrator. This closes the race window without reintroducing a cross-host migration write-race.

Notes

  • The AddPooledDbContextFactory bare UseNpgsql in AddOpenShockDB is dead config (TryAdd means the enum-mapped pool options win), so it was not the cause — left untouched here.
  • Compatible with the integration test harness, which migrates the fresh DB before booting the host (gate sees 0 pending → proceeds immediately).

Open in Stage

… context

The Cron host does not own migrations (the API is the sole migrator), but its
OpenShockContext binds Postgres enum types (email_status, email_type, ...) to CLR
enums by name via MapEnum<T>(). Npgsql resolves those names against the type
catalog it loads on the pooled data source's first connection and caches for the
life of the process. If that first connection happens before the migrator has
created a newly-added enum, every query using it fails permanently with
'data type name ... could not be found in the types that were loaded by Npgsql'
until the process restarts - which is exactly what hit the EmailOutboxDeliveryJob
claim query during the outbox rollout.

Add WaitForOpenShockSchemaReady (polls GetPendingMigrationsAsync with capped
backoff and a timeout) and call it in the Cron host right after the common
middleware, before Hangfire or any job opens the pooled context. Cron performs no
schema writes; the API stays the sole migrator.
Copilot AI review requested due to automatic review settings July 1, 2026 10:32
@stage-review

stage-review Bot commented Jul 1, 2026

Copy link
Copy Markdown

Ready to review this PR? Stage has broken it down into 2 individual chapters for you:

Title
1 Implement schema readiness polling utility
2 Gate Cron startup on schema readiness
Open in Stage

Chapters generated by Stage for commit 9b2472c on Jul 1, 2026 10:32am UTC.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a startup gate to the Cron host so it won’t open an enum-mapped OpenShockContext pool until the database schema (migrations) is fully applied by the API host, preventing Npgsql from permanently caching “missing” Postgres enum types during deploy races.

Changes:

  • Introduces WaitForOpenShockSchemaReady(DatabaseOptions, TimeSpan?) which polls GetPendingMigrationsAsync() with capped exponential backoff and a timeout.
  • Calls the new gate in Cron/Program.cs immediately after UseCommonOpenShockMiddleware() and before Hangfire/job execution can touch the pooled context.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
Cron/Program.cs Blocks Cron startup until schema is migration-complete before any job execution can open/use the pooled EF Core context.
Common/OpenShockMiddlewareHelper.cs Adds a reusable schema-readiness wait helper for non-migrator hosts to avoid enum type-catalog caching races.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@hhvrc
hhvrc merged commit 32f925b into develop Jul 1, 2026
32 of 33 checks passed
@hhvrc
hhvrc deleted the fix/cron-schema-ready-gate branch July 1, 2026 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants