Skip to content

Self-hosted runner (macOS/ARM64): Runner.Worker deadlocks at job startup right after creating the results WebSocket (ingest.sock), holding the runner busy #4558

Description

@mfortunas

Describe the bug

On self-hosted macOS/ARM64 runners, Runner.Worker intermittently deadlocks at job startup, immediately after opening the results/live-log WebSocket. The worker process stays alive, keeps renewing the job lease (so GitHub shows the runner busy), but never spawns the first step — blocking all queued jobs for hours until the process is killed.

Runner Version and Platform

  • Runner: 2.335.1 (also reproduced on 2.334.0 — see below)
  • OS: macOS (Darwin 25.5.0), Apple Silicon (M4), self-hosted, running as a launchd service
  • Trigger: normal pull_request CI (matrix of Rust/gate jobs)

What's the expected behavior?

The worker sets up the results WebSocket and proceeds to run the job steps.

Symptom / evidence

The Worker_*.log freezes right after:

[ResultServer] Creating websocket client ...wss://results-receiver.actions.githubusercontent.com/_ws/ingest.sock
...
[JobServerQueue] Stop aggressive process web console line queue.

Then: no further log lines, no child process spawned (checkout never runs), ~0% CPU, for hours. JobDispatcher keeps logging Successfully renew job ..., so the runner reports busy=true to GitHub and no new jobs are dispatched to it.

sample <Runner.Worker pid> (macOS) shows all ~17 threads parked on .NET synchronization primitives and zero threads in socket/network I/O:

main thread: ... RunMain -> [managed] -> ObjectNative::WaitTimeout -> SyncBlock::Wait
             -> Thread::Block -> _pthread_cond_wait -> __psynch_cvwait
aggregate:   ~10x InternalWaitForMultipleObjectsEx, 4x WaitHandleNative::CorWaitOneNative,
             3x CorWaitOnePrioritizedNative, 2x ObjectNative::WaitTimeout - 0x SSLRead/recv

i.e. an internal deadlock, not a network stall (a network hang would show a thread blocked in a socket read).

Ruled out during investigation

  • Network: basic connectivity is fine; a curl WebSocket upgrade to results-receiver.../_ws/ingest.sock returns HTTP 401 in ~0.5s. Thread sample shows no thread doing network I/O.
  • Runner version: downgrading to 2.334.0 hangs identically at the same WebSocket point (both versions contain the same ResultServer/WebSocket code).
  • Thread-pool starvation: DOTNET_ThreadPool_ForceMinWorkerThreads=64 (verified present in the Worker's env) has no effect.

Behaves intermittently — some jobs complete; during bad spells nearly every job hangs at this point. Looks related to #2140 and the (closed) #1496.

Questions

  1. Is there a known deadlock in the results/live-log WebSocket setup path?
  2. Is there a client-side way to disable the live-log WebSocket feed (fall back to HTTP batch upload) as a workaround?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions