fix(runtime): bound worker joins during scheduler teardown

[gertybotbot]

Closes #2508.

The defect

teardown_workers (hew-runtime/src/scheduler.rs) hard-joined every worker thread with no timeout:

if let Some(h) = handle.take() {
    if h.join().is_err() { ... }
}

A worker genuinely parked in a blocking syscall (e.g. accept()) never reaches the park-recheck, so join() never returns and shutdown hangs forever.

Scope correction

The issue says this "does NOT affect any compiled .hew binary today" and is latent hardening for the not-yet-wired host-embedder path. That understates it.

teardown_workers has three callers, and one is hew_sched_shutdown — which codegen emits directly into native main for supervisor programs (hew-codegen-rs/src/llvm.rs, emit_immediate_shutdown_epilogue, alongside the hew_sched_shutdown ABI declaration in runtime_abi.rs). Supervisor programs deliberately bypass the generic idle drain and go straight to closing and joining workers, so they reach the unbounded join today. hew_runtime_cleanup being unreachable is true but is not the only route.

The fix is the same either way; the reachability is just wider than filed.

The fix

Poll is_finished() against a deadline, and join() only once the thread is known finished so the join itself cannot block.

Two deliberate choices:

• One deadline for the whole worker set, not per-handle — N workers each parked in a blocking syscall must not multiply the bound by N. Covered by a dedicated test.
• Detach, do not force-kill. A worker past the deadline may still be touching scheduler-owned memory and there is no safe way to interrupt it, so the handle is leaked via mem::forget and reported on stderr. Bounding shutdown must not introduce a use-after-free; leaking a thread is the fail-closed choice. This mirrors the existing task-scope reaper precedent.

WORKER_JOIN_TIMEOUT is 5s, matching shutdown::DEFAULT_DRAIN_TIMEOUT_MS. By the time teardown runs, the drain phase has already had its own budget, so this is a backstop against a worker that cannot return — not a second drain window. The polling shape follows the house convention in shutdown::drain_until_idle.

The stale doc comment on hew_sched_shutdown (which promised an unconditional join of every worker) is updated to state the bound.

Verification

Three tests, proving discrimination rather than presence:

test	proves
teardown_joins_workers_that_exit	CONTROL — a normally-exiting worker is still joined (its write is visible on return), and teardown returns well before the timeout
teardown_abandons_a_worker_that_never_exits	a stuck worker is bounded and teardown returns
teardown_deadline_is_shared_across_workers	3 stuck workers cost ~1x, not 3x, the timeout

Revert-control: with the join loop reverted to h.join() but the tests kept, teardown_abandons_a_worker_that_never_exits runs past 60s and has to be killed — i.e. it reproduces exactly the hang #2508 describes. With the fix it passes in ~5s. The test is a real discriminator, not a tautology.

Green bare (exit codes read directly, not through a pipe):

• cargo test -p hew-runtime --lib — 2243 passed, RC=0
• cargo clippy -p hew-runtime --all-targets -- -D warnings — RC=0
• cargo fmt -p hew-runtime -- --check — RC=0

The pre-existing spawn_failure_teardown_joins_partial_worker_set_and_drops_scheduler regression test still passes.

Unrelated flake found while verifying

transport::tests::framed_send_to_broken_pipe_fails_closed_without_signal fails intermittently in full-suite runs. I initially suspected my change and ran it down:

• unmodified main, 6 full-suite runs: 5 ok / 1 FAILED — so it is pre-existing and not caused by this PR.
• It passes in isolation on both branches; only concurrent full-suite runs trigger it.
• The assertion failure reports left: 4096 — a fully successful 4096-byte write where -1 was expected. So the send did not hit EPIPE at all: after drop(far) closes the peer fd, a concurrent thread elsewhere in the suite opens a new fd that reuses that number, and the write lands on a live socket. Classic fd-reuse race inherent to the test, not a defect in the SIGPIPE suppression it is meant to guard.

My diff contains no fd-touching code (only sleeps, is_finished(), join, and mem::forget), though the added ~5s of wall-clock does widen the window. Filing separately rather than folding an unrelated fix into this PR.