Describe the bug
On self-hosted macOS/ARM64 runners, Runner.Worker intermittently deadlocks at job startup, immediately after opening the results/live-log WebSocket. The worker process stays alive, keeps renewing the job lease (so GitHub shows the runner busy), but never spawns the first step — blocking all queued jobs for hours until the process is killed.
Runner Version and Platform
- Runner: 2.335.1 (also reproduced on 2.334.0 — see below)
- OS: macOS (Darwin 25.5.0), Apple Silicon (M4), self-hosted, running as a launchd service
- Trigger: normal
pull_request CI (matrix of Rust/gate jobs)
What's the expected behavior?
The worker sets up the results WebSocket and proceeds to run the job steps.
Symptom / evidence
The Worker_*.log freezes right after:
[ResultServer] Creating websocket client ...wss://results-receiver.actions.githubusercontent.com/_ws/ingest.sock
...
[JobServerQueue] Stop aggressive process web console line queue.
Then: no further log lines, no child process spawned (checkout never runs), ~0% CPU, for hours. JobDispatcher keeps logging Successfully renew job ..., so the runner reports busy=true to GitHub and no new jobs are dispatched to it.
sample <Runner.Worker pid> (macOS) shows all ~17 threads parked on .NET synchronization primitives and zero threads in socket/network I/O:
main thread: ... RunMain -> [managed] -> ObjectNative::WaitTimeout -> SyncBlock::Wait
-> Thread::Block -> _pthread_cond_wait -> __psynch_cvwait
aggregate: ~10x InternalWaitForMultipleObjectsEx, 4x WaitHandleNative::CorWaitOneNative,
3x CorWaitOnePrioritizedNative, 2x ObjectNative::WaitTimeout - 0x SSLRead/recv
i.e. an internal deadlock, not a network stall (a network hang would show a thread blocked in a socket read).
Ruled out during investigation
- Network: basic connectivity is fine; a
curl WebSocket upgrade to results-receiver.../_ws/ingest.sock returns HTTP 401 in ~0.5s. Thread sample shows no thread doing network I/O.
- Runner version: downgrading to 2.334.0 hangs identically at the same WebSocket point (both versions contain the same ResultServer/WebSocket code).
- Thread-pool starvation:
DOTNET_ThreadPool_ForceMinWorkerThreads=64 (verified present in the Worker's env) has no effect.
Behaves intermittently — some jobs complete; during bad spells nearly every job hangs at this point. Looks related to #2140 and the (closed) #1496.
Questions
- Is there a known deadlock in the results/live-log WebSocket setup path?
- Is there a client-side way to disable the live-log WebSocket feed (fall back to HTTP batch upload) as a workaround?
Describe the bug
On self-hosted macOS/ARM64 runners,
Runner.Workerintermittently deadlocks at job startup, immediately after opening the results/live-log WebSocket. The worker process stays alive, keeps renewing the job lease (so GitHub shows the runnerbusy), but never spawns the first step — blocking all queued jobs for hours until the process is killed.Runner Version and Platform
pull_requestCI (matrix of Rust/gate jobs)What's the expected behavior?
The worker sets up the results WebSocket and proceeds to run the job steps.
Symptom / evidence
The
Worker_*.logfreezes right after:Then: no further log lines, no child process spawned (checkout never runs), ~0% CPU, for hours.
JobDispatcherkeeps loggingSuccessfully renew job ..., so the runner reportsbusy=trueto GitHub and no new jobs are dispatched to it.sample <Runner.Worker pid>(macOS) shows all ~17 threads parked on .NET synchronization primitives and zero threads in socket/network I/O:i.e. an internal deadlock, not a network stall (a network hang would show a thread blocked in a socket read).
Ruled out during investigation
curlWebSocket upgrade toresults-receiver.../_ws/ingest.sockreturns HTTP 401 in ~0.5s. Thread sample shows no thread doing network I/O.DOTNET_ThreadPool_ForceMinWorkerThreads=64(verified present in the Worker's env) has no effect.Behaves intermittently — some jobs complete; during bad spells nearly every job hangs at this point. Looks related to #2140 and the (closed) #1496.
Questions