Heal stale active rows when a test runner dies mid-run#265
Merged
Conversation
When a test runner is killed mid-job (docker restart, OOM kill, host reboot), nothing flipped its DB row off status='active'. /search kept showing a ghost active entry indefinitely, and /result rendered "Streaming logs · running" over the dead worker's frozen log output — because the queue still keeps the job in its active set until its periodic stall check moves it back to wait (~30 s default), and getState() can't distinguish a real worker from an orphaned lock. Operators saw two jobs apparently running on one test runner after a restart even though per-worker concurrency=1 was intact: one was real, the other was the abandoned ghost. Three fixes close the gap from different sides. A global:stalled listener mirrors the queue back to the DB the moment a stalled job goes back to wait. A 60-second reconcile pass picks up rows the listener missed because the server itself was also restarting when the stall fired. And /result/:id peeks at the job's lock key in Redis so an orphaned-active job renders as "stalled, retrying" immediately instead of waiting 30 seconds for the stall check to catch up. Stalled rows are mirrored as waiting rather than failed, so a successful retry doesn't leave a phantom failure in the 24 h health pill. Co-authored-by: Claude noreply@anthropic.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a test runner is killed mid-job (docker restart, OOM kill, host reboot), nothing flipped its DB row off status='active'. /search
kept showing a ghost active entry indefinitely, and /result rendered "Streaming logs · running" over the dead worker's frozen log
output — because the queue still keeps the job in its active set until its periodic stall check moves it back to wait (~30 s
default), and getState() can't distinguish a real worker from an orphaned lock. Operators saw two jobs apparently running on one test
runner after a restart even though per-worker concurrency=1 was intact: one was real, the other was the abandoned ghost.
Three fixes close the gap from different sides. A global:stalled listener mirrors the queue back to the DB the moment a stalled job
goes back to wait. A 60-second reconcile pass picks up rows the listener missed because the server itself was also restarting when
the stall fired. And /result/:id peeks at the job's lock key in Redis so an orphaned-active job renders as "stalled, retrying"
immediately instead of waiting 30 seconds for the stall check to catch up. Stalled rows are mirrored as waiting rather than failed,
so a successful retry doesn't leave a phantom failure in the 24 h health pill.
Co-authored-by: Claude noreply@anthropic.com