Turn the org backfill into a daily background worker - #146
Open
peetzweg wants to merge 1 commit into
Open
Conversation
Make backfill-orgs a scheduled task that drains the recorded-but-unfilled org queue a bounded slice at a time, instead of a one-off script that has to run to completion. Extract the lock / budget-guard / runtime-cap plumbing that monthly-user-refresh proved out into src/lib/job-runner.ts, and rebuild the backfill on top of it as an engine + store + tests. Per-run caps (--max-requests, --max-runtime-minutes) stop the run cleanly and leave the rest for tomorrow; builtAt is stamped only once every enumerated member is fetched, so a partially-filled org stays queued and resumes from org_members.last_fetched. A 2,800-member org now gets chipped away over several nights rather than blocking the queue. Also raise the member pagination cap for the worker (the request path keeps its own) and report truncation, since three orgs in the queue have more than the 1,000 members the old cap could enumerate. Two reliability fixes found while testing against the dev database: - cleanup now releases the advisory lock first and defensively. A crash mid-teardown skipped pg_advisory_unlock, leaving the lock held by a server-side session that outlived the process, so the next run reported status=locked and did nothing. - pool shutdown can no longer flip the exit code. postgres.js threw asynchronously while tearing sockets down, which would show a completed run as a failed Coolify execution. A run that fails every org without fetching anything now gives up after three and exits non-zero, rather than printing "Done" and looking like success.
|
The preview deployment for peetzweg/commit-history:coolify-bwc8csg4w8gkkgcg8s404cks is ready. 🟢 Open Preview | Open Build Logs | Open Application Logs Last updated at: 2026-07-30 09:52:16 CET |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #107 (epic #106). Makes
backfill-orgsa Coolify scheduled task that drains the recorded-but-unfilled org queue a bounded slice per run, instead of a one-off script that has to run to completion.Why: 17 orgs are recorded but unfilled, and three of them (EpicGames, EddieHubCommunity, zero-to-mastery) have ~7,300 members between them, far more than one run can ever fetch. The old script also silently gave up: a run on 13 July failed every single org in 18 seconds and still printed
Done, which is why they looked filled. As more companies get looked up, this needs to just happen nightly.How: the lock / budget-guard / runtime-cap plumbing from
monthly-user-refreshmoves intosrc/lib/job-runner.ts, and the backfill is rebuilt on it as engine + store + tests. Per-run caps stop cleanly and leave the rest for tomorrow;builtAtis stamped only once every enumerated member is fetched, so a partial org stays queued and resumes offorg_members.last_fetched. Queue fills always beat the (opt-in, default off) staleness rotation, so refreshing old numbers can't starve an org that has none. The one judgment call: the worker gets a higher member-pagination cap than the request path, since a 2,800-member org is exactly the case the request path refuses and this exists to handle.Two reliability fixes surfaced by running it against the dev database, both applied to the monthly worker too: cleanup now releases the advisory lock first and defensively (a crash mid-teardown skipped
pg_advisory_unlockand left it held by a session that outlived the process, so the next run reportedstatus=lockedand did nothing), and pool shutdown can no longer throw its way into flipping the exit code.No migration. 15 new unit tests; verified against the dev DB end to end — filled 3 small orgs for real, resumed a partial mega-org across runs, and confirmed all three mega-orgs now enumerate fully.
Deploy step is manual: add a Coolify scheduled task
0 6 * * *runningnode .output/worker/backfill-orgs.mjs, clear of the monthly user refresh.