Skip to content

Heal stale active rows when a test runner dies mid-run#265

Merged
soulgalore merged 1 commit into
mainfrom
onlinetest-stale-active-reconcile
May 15, 2026
Merged

Heal stale active rows when a test runner dies mid-run#265
soulgalore merged 1 commit into
mainfrom
onlinetest-stale-active-reconcile

Conversation

@soulgalore

Copy link
Copy Markdown
Member

When a test runner is killed mid-job (docker restart, OOM kill, host reboot), nothing flipped its DB row off status='active'. /search
kept showing a ghost active entry indefinitely, and /result rendered "Streaming logs · running" over the dead worker's frozen log
output — because the queue still keeps the job in its active set until its periodic stall check moves it back to wait (~30 s
default), and getState() can't distinguish a real worker from an orphaned lock. Operators saw two jobs apparently running on one test
runner after a restart even though per-worker concurrency=1 was intact: one was real, the other was the abandoned ghost.

Three fixes close the gap from different sides. A global:stalled listener mirrors the queue back to the DB the moment a stalled job
goes back to wait. A 60-second reconcile pass picks up rows the listener missed because the server itself was also restarting when
the stall fired. And /result/:id peeks at the job's lock key in Redis so an orphaned-active job renders as "stalled, retrying"
immediately instead of waiting 30 seconds for the stall check to catch up. Stalled rows are mirrored as waiting rather than failed,
so a successful retry doesn't leave a phantom failure in the 24 h health pill.

Co-authored-by: Claude noreply@anthropic.com

  When a test runner is killed mid-job (docker restart, OOM kill, host reboot), nothing flipped its DB row off status='active'. /search
   kept showing a ghost active entry indefinitely, and /result rendered "Streaming logs · running" over the dead worker's frozen log
  output — because the queue still keeps the job in its active set until its periodic stall check moves it back to wait (~30 s
  default), and getState() can't distinguish a real worker from an orphaned lock. Operators saw two jobs apparently running on one test
   runner after a restart even though per-worker concurrency=1 was intact: one was real, the other was the abandoned ghost.

  Three fixes close the gap from different sides. A global:stalled listener mirrors the queue back to the DB the moment a stalled job
  goes back to wait. A 60-second reconcile pass picks up rows the listener missed because the server itself was also restarting when
  the stall fired. And /result/:id peeks at the job's lock key in Redis so an orphaned-active job renders as "stalled, retrying"
  immediately instead of waiting 30 seconds for the stall check to catch up. Stalled rows are mirrored as waiting rather than failed,
  so a successful retry doesn't leave a phantom failure in the 24 h health pill.

  Co-authored-by: Claude noreply@anthropic.com
@soulgalore
soulgalore merged commit 27732fe into main May 15, 2026
10 checks passed
@soulgalore
soulgalore deleted the onlinetest-stale-active-reconcile branch May 15, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant