fix(hive): stop workers from getting silently stuck (#1811) - #1812
Open
konard wants to merge 10 commits into
Open
fix(hive): stop workers from getting silently stuck (#1811)#1812konard wants to merge 10 commits into
konard wants to merge 10 commits into
Conversation
Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: #1811
Add a deep-dive case study at docs/case-studies/issue-1811/ documenting the silent stall observed in a 5-issue hive run: - README.md (TL;DR + index) - timeline.md (reconstructed event sequence with log-line citations) - requirements.md (R1-R5 + non-goals) - root-causes.md (RC1-RC5: gh has no default network timeout, wrapDollarWithGhRetry has no timeout layer, hive worker has no inactivity watchdog, verbose mode does not log shell commands) - solutions.md (implementation map, components considered, test plan) - upstream-issue-draft.md (cli/cli issue draft with reproduction) - logs/hive-issue-1811.txt (8.4MB failing-run log; .txt extension to bypass repo .gitignore that excludes *.log) The case study is the prerequisite analysis for the code changes that follow on this branch (PR #1812).
The `.gitignore` rule `*.log` was silently excluding the reproduction log from git tracking. Rename to `hive-issue-1811.txt` so the log ships with the case study, and update the three references in requirements.md, root-causes.md and timeline.md accordingly. Issue #1811.
Closes the silent-stall reported in #1811. Three layered fixes: * RC2: `wrapDollarWithGhRetry` now accepts a per-call `timeoutMs` (default `timeouts.ghApiMs`, 15s, env `HIVE_MIND_GH_API_TIMEOUT_MS`). Each `gh` invocation is raced against the timeout; on expiry the spawned child is SIGTERMed via command-stream's `signal` option (with safe fallback when an older command-stream rejects the options form). The timeout is exposed as a typed `GhTimeoutError` that feeds the existing transient-error budget so legitimately slow calls still retry. * RC1: `verifyResults` now calls `gh api user` (and `gh pr list`, `gh search prs`) under that timeout and degrades gracefully — if user lookup times out, the branch-only PR search continues with `currentUser = null`. Per-call shell tracing is emitted under `--verbose` so future hangs are diagnosable from the log alone. * RC3: `hive.mjs:worker()` gains a parent-side inactivity watchdog. Every stdout/stderr line marks the child as alive; a 1s interval warns when `--worker-inactivity-warn-seconds` is exceeded (default 300s) and SIGTERMs (then SIGKILLs after a 10s grace) when `--worker-inactivity-kill-seconds` is set. Verbose mode emits an earlier heartbeat between 60s and the warn threshold. New config: * `timeouts.ghApiMs` (env `HIVE_MIND_GH_API_TIMEOUT_MS`, default 15000) * `workers.inactivityWarnMs`, `workers.inactivityKillMs`, `workers.killGraceMs` (env-overridable) New CLI flags: * `--worker-inactivity-warn-seconds` (default 300) * `--worker-inactivity-kill-seconds` (default 0 = disabled) See `docs/case-studies/issue-1811/` for the full root-cause analysis.
…outError Adds 12 new unit tests for the issue #1811 timeout machinery in github-rate-limit.lib.mjs and tightens the production wrapper: - callWithTimeout: silence the loser of the Promise.race so its rejection never surfaces as an unhandled rejection; remove the unref() on the inner timer (it caused Node to exit before the timeout could fire in synthetic tests). - wrapDollarWithGhRetry: make the options-form invocation (`dollar({ signal })`) opt-in via `supportsOptionsForm: true`. This keeps backward-compat for naive `$` test fakes that don't distinguish the two calling conventions, while solve.results.lib.mjs opts in so command-stream's signal-driven cancellation still propagates to the spawned `gh` child. New tests: - callWithTimeout (3): resolves/rejects/disabled. - ghWithRateLimitRetry with timeoutMs (2): retry budget, retryOnTimeout=false. - wrapDollarWithGhRetry options form (7): per-call timeoutMs, wrapper defaultTimeoutMs, defaultTimeoutMs=0, verboseLog, $.gh({...}), supportsOptionsForm on/off. All 39 tests pass.
Move the inline 70-line watchdog from hive.mjs into a standalone src/hive-worker-watchdog.lib.mjs with an injectable clock and onEvent callback, so the warn/heartbeat/SIGTERM/SIGKILL state machine can be unit-tested without spawning real child processes. Covered by 8 new tests in tests/test-hive-worker-watchdog-1811.mjs. Refs #1811.
Document the gh timeout + worker inactivity watchdog as a minor release and bump the version accordingly.
konard
changed the title
konard
marked this pull request as ready for review
May 16, 2026 23:03
CI feedback on #1811: - check-file-line-limits failed: hive.mjs was 1537 lines (limit 1500). Move the per-worker watchdog wiring into wireHiveWorkerWatchdog() inside hive-worker-watchdog.lib.mjs and auto-attach the stdout/stderr/ close/error listeners there. hive.mjs is now 1500 lines, net +2 vs main. - Check for Manual Version Changes failed: bumps are managed by the release workflow via the changeset, so revert package.json 1.73.0 -> 1.72.0. - Prettier format check failed on two case-study docs and the new watchdog helper; re-format.
The opts-form `$({ cwd: tempDir })\`git ...\`` invocations in
solve.results.lib.mjs#cleanupClaudeFile were silently dropping `cwd`
because the wrapper introduced in #1811 treated every key in the
per-call options object as a gh-retry option. Git commands then ran in
the wrong directory, breaking test-issue-1791-gitkeep-cleanup test #3
on this branch.
Now split per-call options into wrapper-owned keys (timeoutMs,
retryOnTimeout, label, log, ...) and dollar-owned keys (cwd, env,
stdin, signal, ...), and forward the latter to the underlying dollar
via its own options-form when it supports it. Falls back to the bare
tagged-template invocation when dollar doesn't accept the options form.
Adds 4 regression tests covering: cwd forwarding for non-gh tagged
templates, cwd forwarding for gh tagged templates without timeout, cwd
merged with signal for gh with supportsOptionsForm + timeoutMs, and
that wrapper-only options (timeoutMs) don't leak into dollar options.
Contributor
Author
Working session summaryPR is open, ready, mergeable. CI in progress. The background task will notify on completion. This summary was automatically extracted from the AI working session output. |
Contributor
Author
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $24.245940📊 Context and tokens usage:Claude Opus 4.7: (8 sub-sessions)
Total: (28.0K new + 742.0K cache writes + 27.7M cache reads) input tokens, 223.9K output tokens, $24.245940 cost 🤖 Models used:
📎 Log file uploaded as Gist (11816KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
Contributor
Author
✅ Ready to mergeThis pull request is now ready to be merged:
Monitored by hive-mind with --auto-restart-until-mergeable flag |
This reverts commit f50f1f2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1811 — hive workers occasionally stalled inside
verifyResultswith no progress, no failure, and no recovery. The root cause was a hunggh api usercall (the GitHub CLI has no default HTTP timeout). The hive parent had no way to notice "child is alive but quiet," so the stuck worker would block its slot indefinitely.This PR adds two complementary safety nets and the supporting test + documentation work.
1.
ghcall-level timeout (src/github-rate-limit.lib.mjs)GhTimeoutErrorandcallWithTimeout(fn, ms, signal)helper that races a promise against asetTimeoutand aborts the underlyingAbortControllerwhen the timer wins (the loser's rejection is silenced so we don't leak unhandled rejections).ghWithRateLimitRetry/wrapDollarWithGhRetrynow accept a per-calltimeoutMsand a wrapper-leveldefaultTimeoutMs.wrapDollarWithGhRetrycan opt in (supportsOptionsForm: true) to the dual-form$({ signal })API exposed bycommand-stream, so a timed-out call actually SIGTERMs the spawnedghchild instead of just rejecting the JS promise.solve.results.lib.mjsopts in; tests with naive$fakes do not, preserving backward-compatible single-call semantics.GH_TIMEOUT_SECONDSenv /--gh-timeout-secondsCLI flag, 90s default. Set to0to disable.2. Parent-side worker inactivity watchdog (
src/hive-worker-watchdog.lib.mjs)hive.mjswires it into every spawned worker; stdout/stderr lines callwatchdog.markActivity(line).hive:--worker-inactivity-warn-seconds(default300) — log a warning when a worker emits nothing for N seconds.--worker-inactivity-kill-seconds(default0, disabled) — SIGTERM the worker, then SIGKILL after a 10s grace period.warnMs.3. Case study + upstream report
Full root-cause walkthrough, the actual 90k-line stuck-hive log, timeline, and a draft upstream report for the
ghCLI live underdocs/case-studies/issue-1811/.Files
src/github-rate-limit.lib.mjs—GhTimeoutError,callWithTimeout,timeoutMs/defaultTimeoutMs/supportsOptionsFormplumbing.src/hive-worker-watchdog.lib.mjs(new) — pure helper, injectablenowclock,onEventcallback (kind ∈ {warn, heartbeat, sigterm, sigkill}).src/hive.mjs— uses the watchdog for each spawned worker; pipes events into the existinglog()channel.src/hive.config.lib.mjs— registers the new--worker-inactivity-*flags and adds them toHIVE_ONLY_OPTION_NAMES.src/solve.results.lib.mjs— opts in to the dual-form$API soverifyResultscan actually be timed out.src/config.lib.mjs—gh-timeout-secondsflag +GH_TIMEOUT_SECONDSenv wiring.tests/github-rate-limit.test.mjs— 12 new tests (39 total) coveringGhTimeoutError,callWithTimeout, per-call timeouts, default timeouts, and the opt-in options form.tests/test-hive-worker-watchdog-1811.mjs(new) — 8 tests for the watchdog state machine using a virtual clock and a fake child..changeset/issue-1811-stuck-task-watchdog.md— minor bump to1.73.0.docs/case-studies/issue-1811/**— README, requirements, timeline, root causes, solutions, upstream draft, and the raw hive log.Test plan
node tests/github-rate-limit.test.mjs— 39 passed, 0 failed.node tests/test-hive-worker-watchdog-1811.mjs— 8 passed, 0 failed.node --checkfor every modified.mjs.--worker-inactivity-warn-seconds 10 --worker-inactivity-kill-seconds 30against a worker that getskill -STOP-ed; confirm warning then SIGTERM/SIGKILL fire.