Skip to content

fix(daemon): stop killing live coding agents when the daemon can't report its sessions - #13928

Open
nwparker wants to merge 23 commits into
mainfrom
nwparker/orca-lost-new-moon
Open

fix(daemon): stop killing live coding agents when the daemon can't report its sessions#13928
nwparker wants to merge 23 commits into
mainfrom
nwparker/orca-lost-new-moon

Conversation

@nwparker

@nwparker nwparker commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What broke, in plain terms

Orca doesn't run your terminals inside the app. It runs them inside a separate helper process called the daemon. That's deliberate — it's why you can quit and reopen Orca and your agents are still there.

The catch is that the daemon owns those terminals completely. The file handles for every PTY live inside that one process, and there is no mechanism to hand them to a different process. So the rule is simple:

If the daemon dies, every coding agent it hosts dies with it. There is no recovery. Only the scrollback text is saved — and it gets replayed into a brand-new, empty shell.

Now, sometimes Orca does need to replace the daemon. Before it does, it asks the daemon a reasonable question: "how many live sessions do you have?" If the answer is "more than zero", it backs off, because replacing it would destroy your work.

That question can come back three ways:

Answer Meaning
5 five live sessions — don't kill it
0 nothing to lose — safe to kill
null couldn't get an answer

The bug is what happened on null. The code was:

if (liveSessionCount !== null && liveSessionCount > 0) {
  return preserveDaemon()   // safe
}
// null falls through to:
await killStaleDaemon(...)  // SIGTERM, then SIGKILL after 3s

null fell through into the kill.

Why that's wrong: "I couldn't ask you how many terminals you have" is not the same as "you have no terminals." A daemon that's too busy to answer is still busy hosting. It's the difference between someone not picking up the phone and someone being dead — and we were holding a funeral either way.

And this is not hypothetical. To even reach this code, the daemon has to have already failed a health check — which mostly happens when the machine is under heavy load. Heavy load is also exactly when a busy daemon can't answer within its 3-second budget. The condition that triggers the check is the same condition that makes the check come back blank.

How it looked to the user

You click into a worktree. Your agent is either gone entirely, or it "comes back" by typing claude --resume <id> into a fresh shell. It looks like a restore. It isn't — the original process was killed and everything it hadn't written down is gone.

From a real incident:

01:07:42.052Z  pid 84862  shutdown  reason=SIGTERM      <- daemon holding every agent
01:07:42.436Z  pid 35170  startup   protocolVersion=32  <- replacement, same socket
01:07:42.616Z  session-created  global-floating-terminal@@cbbcaf41
01:07:49.181Z  session-created  .../lint-broom@@e71d60f4
01:07:54.565Z  session-created  .../bug-997-1n-color-scheme...
01:08:00.356Z  session-created  .../perf-renderer-bound-agent-status...

Every line says session-created, never session-attached — brand-new shells. The daemon died in the same second as a sidebar_worktree_activate breadcrumb, i.e. the moment the user clicked into the worktree. The app itself was never restarted; the new daemon is a child of the already-running app process.

Two more details that made this hard to catch:

  1. probeSocket has a 1-second timeout. If it misses, the 11-retry grace loop is skipped entirely and the kill lands in ~4 seconds instead of ~60.
  2. In that fast path, nothing was logged at all. The guard was if (liveSessionCount !== null || graceRetry > 0 || health === 'rejected') — all three false. The daemon was killed silently, which is why nothing in the logs explained it.

Which change introduced this

Two commits, and it's a trade-off resolved the wrong way rather than a careless slip.

Origin — 0b19195fd9b (#5230), 2026-06-11. This wrote the predicate, with its reasoning in the comment:

"Only a verified non-empty list preserves: a daemon that cannot even list sessions cannot serve terminals, and replacing it is the only recovery."

The first half is the error. A daemon that can't list sessions is still hosting them — listing and serving are different capabilities. The same author had written the two sibling branches two weeks earlier (1371402bc6f #2974, 5bfa1413a53 #4330) and both preserve on null, with the comment "live session state could not be verified". This branch diverged from the codebase's own convention.

Reintroduction — 840d3277d1d (#8697, fixing #8689), 2026-07-14. A blanket preserve-on-null had been added four days earlier by #7214 (to stop a Windows update-relaunch killing live terminals) and was masking the defect. #8689 replaced it with the bounded 11-retry grace loop — correctly stopping a permanently wedged daemon from being preserved forever — but that restored kill-on-unknown for anything still unverifiable after ~60s. It left the weak predicate verbatim; it did not author it.

This history is why the fix needed a new signal rather than a flipped boolean. From the launcher's vantage point, "wedged with live agents" and "wedged with nothing" are literally identical: null count, connectable socket. Flipping null to preserve would just re-break #8689.

The fix

Give the question a name, answer it once, and answer it before anything is destroyed.

daemon-occupancy.ts (new)resolveDaemonOccupancy() returns occupied | empty | unknown. It asks the daemon over IPC first, where a reply is authoritative both ways. Only if that fails does it consult the OS process table, and then only to raise the answer to occupied. The table can prove work exists; it can never prove absence. unknown is the residual, and it is total — every unanswerable question lands there.

daemon-live-pty-evidence.ts (new) — the out-of-band read, which never touches the daemon socket, because the socket is what already failed. Details that turned out to matter:

  • Descendants, not children. macOS wraps every shell in login(1) for TCC, so the agent is a grandchild at best.
  • Only session leaders count. forkpty() calls setsid(), which the daemon's plain subprocesses never do.
  • Except the daemon's own probe PTYs, which forkpty also makes session leaders — excluded by program basename with an exact argv tail.
  • Zombies don't count. A wedged daemon can't reap, so its exited agents linger as <defunct> and would read as running — a false positive correlated with the wedge itself.
  • A stranded macOS login wrapper doesn't count. The TCC wrapper can outlive the shell it wrapped (macOS: TCC login-shell wrapper leaks PTYs (session-kill-failed, session-closed never fires) #13764), leaving a session leader hosting nothing — and on hosts where those accumulate, counting them would hold a daemon whose sessions had all ended.
  • A root never observed is unknown, not empty. A walk that never saw the daemon reports zero descendants for a process it never examined.
  • Emptiness needs two agreeing reads. A terminal contributes exactly one session leader — on macOS the login wrapper, since the shell beneath it is S+, not Ss — so a single snapshot cannot tell a wrapper whose shell has gone from one whose shell has not yet appeared. Emptiness is the answer that authorizes a kill, so it pays for a confirming read; owns-live-ptys does not.
  • If the uncached read blows its deadline, the TTL-cached table is used rather than answering blind. Every agent pane already drives that reader, so the host with the most agents to lose is the likeliest to time out on queueing alone.

Windows abstains entirely. The POSIX signal is a property only a hosted terminal has — forkpty makes it a session leader — and Windows has no equivalent, so the branch that lived there could only count descendants. That reads a wedged daemon's orphaned conpty hosts as live work: ClosePseudoConsole runs on the daemon's own JS thread, so a daemon too wedged to answer is too wedged to reap them, and they accumulate exactly when this runs. Rather than add a sixth exclusion, the path is deleted: Windows answers 'unknown' and keeps the behaviour it has on main.

Grace is sized per platform. main gave a wedged daemon roughly a minute to recover (a non-shared 5s connect budget across 12 probes). Bounding those probes made the launch far cheaper but shortened that window, which on POSIX is a good trade — evidence covers what the window no longer does — and on Windows was not, because nothing covers it there. Windows therefore gets the larger share of the launch budget, and neither number is a guess: both are derived in daemon-launch-budget.test.ts from what each platform actually spends.

The launcher decides before it destroys. killStaleDaemon is back to being purely "make this pid go away" — no policy, no options, nothing a future caller can forget to disable, and Manage Sessions → Restart cannot be vetoed because there is no veto to hit.

held is a real outcome. A daemon that owns live terminals but cannot complete a handshake is kept without adoption, without a lease, and without a replacement forked beside it. Both daemons that reach it — rejected (answered and refused) and one whose count only the process table could supply (nothing answered at all) — are gated on the property, not on either of its causes.

Nothing may answer for a session it does not own. write/resize/shutdown/sendSignal now route through the same fence attach already had. Previously an unrouted id fell through to the in-process fallback, whose shutdown returns silently for an id it has never heard of — so closing a held pane reported success while the agent kept running.

The escape hatch stays open

  • Only positive evidence preserves. unknown still replaces, so a permanently wedged daemon with nothing to lose is still replaced ([Bug]: DaemonProtocolError: Hello response timed out #8689).
  • The grace window is bounded by wall clock, sized to fit under the 60s startup fail-open — overrunning it would trade a wedged daemon for no daemon and a restartDaemon() that throws.
  • Manage Sessions → Restart still kills.

Proof

pnpm run test:repro:daemon-replacement-live-agent-pty-preservation — real daemon, real PTYs, SIGSTOP as the wedge (socket still accepting, no RPC ever answered — the case the code itself calls out).

Phase 1, the danger is real:

phase 1: checkDaemonHealth() = 'unreachable'
phase 1: probeSocketConnect() = 'connected', endpointIsProvenDead() = false
phase 1: live session count over IPC = null
phase 1: killStaleDaemon() = {"killed":true,"liveOwnerSurvived":false}
phase 1: agent PTY pid 28871 is GONE (ps: no such process)
phase 1: agent PTY pid 29283 is GONE (ps: no such process)

Phase 2, the decision protects it:

phase 2: readVerifiedDaemonPid() = pid 34018 (identity verified: cmdline + start time)
phase 2: resolveDaemonOccupancy() = {"state":"occupied","liveSessions":null}
phase 2: daemon 34018 is STILL ALIVE (ps stat 'Ts' — T = stopped, not killed)
phase 2: agent PTY pid 34076 is ALIVE
phase 2: agent PTY pid 34599 is ALIVE
phase 2: after SIGCONT the daemon is still running — no signal was ever delivered to it
phase 2: resumed daemon reports checkDaemonHealth() = 'healthy', live sessions over IPC = 2

That last line is the whole argument: the wedge was transient. Given the chance, the daemon came back healthy with both sessions intact. Under the old code those agents were already dead.

Phase 3 statically pins that the launcher raises evidence only from a verified pid and returns a held handle before any kill — and says explicitly that it proves source ordering, not execution.

src/main/daemon: 1461 passing. tsc --noEmit: 0 errors. oxlint + oxfmt clean.

What six rounds of adversarial review caught

Recorded because the near-misses are the useful part, and because several were mine.

  1. The fix silently did nothing. Two optional trailing params; the options object landed in the test-hook slot. Every unit test passed — they asserted the same wrong call shape. The real-process script caught it. The signature became one bag, so the mistake is now a type error.
  2. The fix was briefly worse than the bug. Preserving routed through an adoption handshake a wedged daemon cannot answer → init aborted → no spawner → the Restart button my own warning recommended was dead.
  3. Wrong polarity. It was a veto bolted onto a kill decision, so every unenumerated state still ended in a kill — which is why each round found a new one. Rewritten so the decision is made before the kill and unknown never licenses one. Mutation testing also found four survivors, including one where the whole fix could ship as a no-op.
  4. The evidence proved the wrong thing. The session-leader filter counted the daemon's own probe PTYs — its comment's premise refuted its exclusion list two lines down. A daemon hosting nothing could be held on its stuck probe, and dropping our authenticated pair then let it retire.
  5. One question, three answers, two wrong. held had been added as a fifth branch rather than the classification the others route through. Also: the grace budget was a comment with a Date.now() beside it — one probe could cost 50s against a 5s assumption.
  6. Closing a held pane lied. Fixed — and the error type is load-bearing: pty:kill treats "Session not found" as proof the pty is gone and synthesizes an exit, so the obvious error would have reproduced the lie one layer down.

Behaviour changes worth knowing about

Both follow from the change being honest where it previously was not.

  • Closing a pane whose daemon is unreachable now fails instead of appearing to succeed. Previously the in-process fallback answered on the daemon's behalf: the pane vanished and the agent kept running as an orphan that outlived the app. It now reports that the host cannot be reached and keeps ownership so the close can be retried.
  • Deleting a worktree whose PTYs belong to a held daemon now blocks with the existing Force Delete hint, for the same reason — the teardown can no longer be told a stop happened that did not. Correct, but visible, and worth a release note.

Known residuals

  • Windows loses grace relative to main. It had ~60s and now has 26s, bounded by the startup fail-open. A Windows wedge lasting 26–60s is now replaced where main would have adopted it. That is a real, deliberate narrowing of a regression the earlier 12s budget made much worse, and the trade is written down rather than incidental.
  • unknown still replaces. If the daemon cannot answer and the process table cannot be read, behaviour is unchanged from today. Closing that would re-break [Bug]: DaemonProtocolError: Hello response timed out #8689; it needs its own decision.
  • unknown is indistinguishable from "never tried" in the field. A silently broken probe reports the same value as a genuinely blind one, with no telemetry separating them.
  • Nothing transitions out of held for existing sessions within a process lifetime; recovery comes on the next launch or via Restart.
  • The repro script is not a CI gate. No workflow runs test:repro:*, yet it is the only thing exercising the production wiring end to end — and it is what caught the first two defects above.
  • The launch budget is enforced as an invariant over declared constants, not measured end to end. daemon-launch-budget.test.ts sums the health check, grace window, the probe that always overruns a loop-entry ceiling, and the evidence read on both platforms, and requires headroom for the kill ladder and fork. It would still miss a cost nobody declared.
  • One review follow-up deliberately declined. The endpoint-occupied catch was not given the same held fallback as the failed-health path. That path arrives with occupancy unknown or empty, so holding there would swallow a real launch failure to protect nothing — I implemented it, watched it hide a fork error, and reverted.
  • Not addressed: the upstream trigger. A transient client disconnect calls respawn('daemon_died') → full re-ensure, which is what runs this decision mid-session. daemon-init.ts already concedes the app cannot tell wedged from dead there.

Alignment with existing doctrine

src/main/daemon/AGENTS.md already states the rule this branch broke:

Never collapse "can't tell" into "dead." ... A timeout or EPERM proves nothing and must decline — treating it as death deletes an endpoint still serving every terminal on the host.

That was written about the socket path. The session-count gate broke the same principle one layer up, with the same consequence. The codebase also already had the correct three-valued primitive — probeSocketConnect, whose own doc says "absence of proof is not proof of death" — and this file was using a lossy boolean copy of it that returned false on timeout, and unconditionally on Windows named pipes.

nwparker and others added 3 commits August 11, 2026 19:09
…PTYs

A daemon too busy to answer listSessions was indistinguishable from a dead
one: getAliveDaemonSessionCount() returns null ("could not verify"), the
preserve gate required `!== null && > 0`, so the run fell through to
killStaleDaemon() and every running coding agent died with it. The sibling
replace branches all preserve on null; this one alone collapsed "can't tell"
into "empty", which src/main/daemon/AGENTS.md already forbids.

Give the decision an out-of-band second opinion. inspectDaemonPtyOwnership()
reads the OS process table — never the daemon socket, which is exactly what
failed — and reports whether the daemon's own process still has live PTY
descendants. Under preserveWhenOwningLivePtys, that evidence vetoes the
signal and the launcher adopts the daemon in degraded mode instead.

The veto is opt-in so it cannot make a daemon unkillable: only the
failed_health_check path enables it. Manage Sessions -> Restart still kills.
Only positive evidence preserves, so a wedged daemon with nothing to lose is
still replaced (#8689).

Also stop replacing silently: the verdict now prints on the post-kill truth,
which stays quiet on a cold start because nothing was killed.

Co-authored-by: Orca <help@stably.ai>
Adversarial review found the veto's own success path could not complete
against the daemon it exists to protect. Preserving routed through
holdDaemonAdoptionLease(), which opens a hello — the exact operation a
wedged daemon cannot answer — so it threw, aborted initDaemonPtyProvider,
and left no spawner. restartDaemon() throws without one, so the user lost
the documented Manage Sessions -> Restart remedy on top of having no
daemon: strictly worse than the data loss being fixed.

A still-listening endpoint means wedged, not gone, so keep a lease-free
handle instead. The lease only cancels the adoption watchdog, which never
fires on a daemon that owns sessions. Degraded mode likewise tolerates a
lease and a session discovery it cannot complete.

Three more from the same review:

- The veto keyed on reason === 'failed_health_check', but a daemon that
  answered listSessions with 0 lands in that same branch and must stay
  replaceable. Key on liveSessionCount === null, which is what the option
  actually documents.
- Zombies are not evidence of live work. A wedged daemon cannot reap, so
  its exited agents linger as <defunct> and would read as "still running"
  — a false positive correlated with the wedge itself. Enumerate through
  the process table's stat column and exclude them; sample twice so a
  resolver probe or health-check shell cannot masquerade as an agent.
- Restore the "did anything answer?" log guard alongside the confirmed
  kill, so a daemon that self-retires before the kill is still announced.

Co-authored-by: Orca <help@stably.ai>
Mutation testing found four survivors — changes that break the fix while
every test stays green:

- Swapping the POSIX reader to the 500ms-cached one passed. It is not just
  a staleness hazard: inside the TTL both sampling attempts receive the same
  array, collapsing the two-sample confirmation to one. Pin the fresh reader.
- Replacing killStaleDaemon's default inspector with one that never reports
  live PTYs — the veto disabled in production — passed, because every veto
  test injects the hook. Exercise the real seam.
- Adding the veto to cleanupDaemonForProtocol passed, which is verbatim the
  failure its own doc warns about: a user-initiated restart of a daemon
  owning live PTYs would refuse, then throw. Pin that call's arity.
- The ppid-cycle fixture put the cycle outside the daemon's subtree, so the
  walk never entered it and deleting the visited guard passed.

Also bound the Windows enumeration, which had no budget of its own: two CIM
queries with a wmic fallback can stall the launch path for tens of seconds.
Blind is a safe answer there; hanging is not.

Co-authored-by: Orca <help@stably.ai>
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The change adds platform-specific daemon PTY ownership inspection with descendant traversal, filtering, repeated sampling, and timeout handling. Daemon occupancy resolution combines IPC session counts with verified process evidence. Daemon initialization preserves occupied daemons, holds unverifiable daemons, and supports degraded launch modes within a bounded grace period. Tests cover these paths. A POSIX reproduction command validates guarded and unguarded replacement with real processes.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The description explains the problem, solution, rationale, testing, and residual risks, but omits required Linked Issue and Visual Proof sections and does not complete Review or Checklist. Add the issue reference, provide Visual Proof or N/A with a reason, and complete the Review and Checklist sections.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly states the primary daemon-replacement fix: preventing live agents from being killed when session reporting fails.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
src/main/daemon/daemon-init.test.ts (1)

3245-3291: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add coverage for the endpoint-died branch of the new adoption fallback.

This test covers the case where probeSocket succeeds and the launcher returns a lease-free degraded handle. The new catch block in daemon-init.ts has a second branch: when probeSocket returns false, it throws DaemonEndpointOwnershipError because the endpoint is genuinely free.

That branch has no test. An inverted probe condition would still pass every test in this cohort, and it would make the launcher return a degraded handle for a daemon that is actually gone.

The fixture here already stages everything needed. Only the probe result changes.

🧪 Proposed additional test
it('fails the launch when the preserved daemon died before adoption', async () => {
  // Why: a free endpoint means the daemon is gone, not wedged. Returning a degraded
  // handle here would strand the app on a daemon that no longer exists.
  const mod = await importFresh()
  await mod.initDaemonPtyProvider()

  const answeringDefault = function MockDaemonClient() {
    return {
      ensureConnected: vi.fn(async () => {}),
      request: vi.fn(async () => ({ sessions: [] })),
      disconnect: vi.fn()
    }
  }
  daemonClientMock.mockImplementation(function MockWedgedDaemonClient() {
    return {
      ensureConnected: vi.fn(async () => {
        throw new Error('Hello response timed out')
      }),
      getDaemonIdentity: vi.fn(readLaunchedDaemonIdentity),
      request: vi.fn(),
      disconnect: vi.fn()
    }
  })
  killStaleDaemonMock.mockResolvedValueOnce({ killed: false, liveOwnerSurvived: true })

  const launcher = spawnerInstances[0].launcher as (
    socketPath: string,
    tokenPath: string
  ) => Promise<{ shutdown(): Promise<void>; mode?: string }>
  checkDaemonHealthMock.mockResolvedValueOnce('unreachable')
  // The endpoint stopped answering between the refusal and the adoption attempt.
  probeSocketExistsMock.mockReturnValue(false)

  try {
    await expect(launcher('/fake/socket', '/fake/token')).rejects.toThrow(
      /could not be confirmed stopped/
    )
    expect(forkMock).not.toHaveBeenCalled()
  } finally {
    daemonClientMock.mockImplementation(answeringDefault)
  }
})
config/scripts/daemon-replacement-live-agent-pty-preservation-repro.mjs (1)

597-606: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Strip comments before parsing the launcher call, to match readOptionsArgumentSlot.

readOptionsArgumentSlot strips comments before calling splitCallArguments, because comments carry commas that corrupt the top-level split. checkLauncherEnablesTheGuard parses the unstripped source with the same splitter.

Today this works, because the call at daemon-init.ts has no comment between killStaleDaemon( and its closing paren. If someone adds a comment inside the argument list, the split breaks and phase 3 reports a wrong slot.

Keep the unstripped source for the line-number calculation on Line 604, and use a stripped copy only for the split.

♻️ Proposed change
 function checkLauncherEnablesTheGuard(optionsSlot) {
   const relativePath = 'src/main/daemon/daemon-init.ts'
   const source = readFileSync(join(repoRoot, relativePath), 'utf8')
+  // Blank out comments in place so offsets and line numbers still line up.
+  const scannable = source
+    .replace(/\/\*[\s\S]*?\*\//g, (m) => m.replace(/[^\n]/g, ' '))
+    .replace(/\/\/[^\n]*/g, (m) => ' '.repeat(m.length))
   const marker = source.indexOf('preserveWhenOwningLivePtys')
   assert(marker !== -1, `${relativePath} never passes preserveWhenOwningLivePtys`)
   const callIndex = source.lastIndexOf('killStaleDaemon(', marker)
   assert(callIndex !== -1, `${relativePath} sets preserveWhenOwningLivePtys outside a call`)
   const line = source.slice(0, callIndex).split('\n').length
-  const args = splitCallArguments(source, callIndex + 'killStaleDaemon'.length)
+  const args = splitCallArguments(scannable, callIndex + 'killStaleDaemon'.length)

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d79afc4c-ab57-478e-b5ea-6f68316b14bf

📥 Commits

Reviewing files that changed from the base of the PR and between 64aec94 and 12a1366.

📒 Files selected for processing (8)
  • config/scripts/daemon-replacement-live-agent-pty-preservation-repro.mjs
  • package.json
  • src/main/daemon/daemon-health.test.ts
  • src/main/daemon/daemon-health.ts
  • src/main/daemon/daemon-init.test.ts
  • src/main/daemon/daemon-init.ts
  • src/main/daemon/daemon-live-pty-evidence.test.ts
  • src/main/daemon/daemon-live-pty-evidence.ts

Comment thread config/scripts/daemon-replacement-live-agent-pty-preservation-repro.mjs Outdated
Comment on lines +96 to +121
async function loadDaemonPrimitives(scratch) {
const esbuild = await import('esbuild')
const entrySource = join(scratch, 'daemon-primitives-entry.ts')
const bundlePath = join(scratch, 'daemon-primitives.mjs')
const daemonDir = join(repoRoot, 'src', 'main', 'daemon')
writeFileSync(
entrySource,
[
`export { checkDaemonHealth, killStaleDaemon } from ${JSON.stringify(join(daemonDir, 'daemon-health'))}`,
`export { inspectDaemonPtyOwnership } from ${JSON.stringify(join(daemonDir, 'daemon-live-pty-evidence'))}`,
`export { getDaemonPidPath, getDaemonSocketPath, getDaemonTokenPath } from ${JSON.stringify(join(daemonDir, 'daemon-spawner'))}`,
`export { DaemonClient } from ${JSON.stringify(join(daemonDir, 'client'))}`,
''
].join('\n')
)
await esbuild.build({
entryPoints: [entrySource],
outfile: bundlePath,
bundle: true,
platform: 'node',
format: 'esm',
packages: 'external',
logLevel: 'silent'
})
return import(pathToFileURL(bundlePath).href)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Check the esbuild declaration and the electron-freeness of the bundled graph.
set -uo pipefail

echo '===== Is esbuild declared? ====='
jq -r '
  {
    dep: (.dependencies.esbuild // "absent"),
    dev: (.devDependencies.esbuild // "absent")
  }' package.json

echo
echo '===== Does the new import edge reach electron? ====='
for f in \
  src/main/daemon/daemon-live-pty-evidence.ts \
  src/main/providers/windows-foreground-process-rows.ts \
  src/shared/process-table-snapshot.ts
do
  if [ -f "$f" ]; then
    echo "--- $f ---"
    rg -nP "^\s*import .*from ['\"]" "$f"
  else
    echo "--- $f (not found; locating) ---"
    fd -t f "$(basename "$f")" --exec rg -nP "^\s*import .*from ['\"]" {}
  fi
done

echo
echo '===== Any electron import in the four bundled entry modules ====='
for n in daemon-health daemon-live-pty-evidence daemon-spawner client; do
  fd -t f "^${n}\.ts$" src/main/daemon --exec rg -nP "from ['\"]electron['\"]" {} \
    && echo "  ^ electron import found in $n"
done

Repository: stablyai/orca

Length of output: 880


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '===== package-manager declarations and script ====='
jq '{
  packageManager,
  engines,
  scripts,
  dependencies,
  devDependencies
}' package.json

echo
echo '===== esbuild references in manifests and lockfiles ====='
rg -n '(^|["[:space:]])esbuild([@"[:space:]]|$)' \
  package.json pnpm-workspace.yaml pnpm-lock.yaml package-lock.json yarn.lock 2>/dev/null || true

echo
echo '===== loader and bundled entry imports ====='
sed -n '1,170p' config/scripts/daemon-replacement-live-agent-pty-preservation-repro.mjs
for f in \
  src/main/daemon/daemon-health.ts \
  src/main/daemon/daemon-live-pty-evidence.ts \
  src/main/daemon/daemon-spawner.ts \
  src/main/daemon/client.ts \
  src/main/providers/windows-foreground-process-rows.ts \
  src/shared/process-table-snapshot.ts
do
  echo "--- $f ---"
  sed -n '1,180p' "$f"
done

echo
echo '===== transitive relative-import graph and package imports ====='
python3 - <<'PY'
import re
from pathlib import Path

entries = [
    Path("src/main/daemon/daemon-health.ts"),
    Path("src/main/daemon/daemon-live-pty-evidence.ts"),
    Path("src/main/daemon/daemon-spawner.ts"),
    Path("src/main/daemon/client.ts"),
]
seen = set()
stack = entries[:]
while stack:
    p = stack.pop()
    if p in seen:
        continue
    seen.add(p)
    if not p.exists():
        print(f"MISSING {p}")
        continue
    text = p.read_text()
    imports = re.findall(r"""(?:import\s+(?:[^'"]+\s+from\s+)?|export\s+(?:[^'"]+\s+from\s+)?|import\s*\()\s*['"]([^'"]+)['"]""", text)
    print(f"{p}:")
    for spec in imports:
        if spec.startswith("."):
            q = (p.parent / spec)
            candidates = [q, q.with_suffix(".ts"), q.with_suffix(".tsx"), q / "index.ts"]
            target = next((x for x in candidates if x.exists()), None)
            print(f"  relative {spec} -> {target or 'MISSING'}")
            if target:
                stack.append(target)
        else:
            print(f"  package {spec}")
print("\n===== electron references in discovered files =====")
for p in sorted(seen):
    if p.exists():
        for i, line in enumerate(p.read_text().splitlines(), 1):
            if re.search(r"""(?:from\s+|import\s*\()\s*['"]electron(?:/[^'"]*)?['"]""", line):
                print(f"{p}:{i}:{line}")
PY

Repository: stablyai/orca

Length of output: 50369


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '===== root importer esbuild entries ====='
python3 - <<'PY'
from pathlib import Path
text = Path("pnpm-lock.yaml").read_text()
start = text.find("importers:")
end = text.find("\npackages:", start)
print(text[start:end][:12000])
PY

echo
echo '===== esbuild lockfile references ====='
rg -n -C 4 '^(\s+)?esbuild(@|:)|esbuild:' pnpm-lock.yaml | head -120

echo
echo '===== complete relative-import graph ====='
python3 - <<'PY'
import re
from pathlib import Path

entries = [
    Path("src/main/daemon/daemon-health.ts"),
    Path("src/main/daemon/daemon-live-pty-evidence.ts"),
    Path("src/main/daemon/daemon-spawner.ts"),
    Path("src/main/daemon/client.ts"),
]
seen = set()
stack = entries[:]
package_imports = []
electron_refs = []

def resolve_import(source, spec):
    base = source.parent / spec
    candidates = [base]
    if base.suffix == "":
        candidates += [
            base.with_suffix(".ts"),
            base.with_suffix(".tsx"),
            base.with_suffix(".js"),
            base.with_suffix(".mjs"),
            base / "index.ts",
        ]
    return next((p for p in candidates if p.exists()), None)

while stack:
    source = stack.pop()
    if source in seen:
        continue
    seen.add(source)
    if not source.exists():
        print(f"MISSING_ENTRY_OR_IMPORT {source}")
        continue

    for line_no, line in enumerate(source.read_text().splitlines(), 1):
        matches = re.findall(
            r"""(?:import\s+(?:[^'"]+\s+from\s+)?|export\s+(?:[^'"]+\s+from\s+)?|import\s*\()\s*['"]([^'"]+)['"]""",
            line,
        )
        for spec in matches:
            if spec == "electron" or spec.startswith("electron/"):
                electron_refs.append((source, line_no, spec))
            if spec.startswith("."):
                target = resolve_import(source, spec)
                if target is None:
                    print(f"UNRESOLVED_RELATIVE {source}:{line_no} {spec}")
                else:
                    stack.append(target)
            else:
                package_imports.append((source, line_no, spec))

print(f"discovered_files={len(seen)}")
print("package_imports:")
for source, line_no, spec in sorted(package_imports):
    print(f"  {source}:{line_no}: {spec}")
print("electron_refs:")
for source, line_no, spec in sorted(electron_refs):
    print(f"  {source}:{line_no}: {spec}")
PY

Repository: stablyai/orca

Length of output: 15269


Declare esbuild in devDependencies. The script imports it directly, but only a transitive lockfile entry exists. The complete bundled graph has no electron import.

…ons that can never be adopted

Round-three review found the veto too eager in three ways, each of which
traded the original data loss for a whole-session degrade or a permanently
daemon-less app.

Evidence was "any non-zombie descendant", justified by re-sampling to weed
out transients. The two samples are taken back to back — one ps fork apart —
so nothing transient is ever weeded out, and a hung helper (often the very
reason the daemon is wedged) reads as an agent. Use the structural signal
instead: a PTY child is a session leader, because forkpty calls setsid, and
no helper the daemon forks ever is. Re-sampling now only retries blindness.

A 'rejected' daemon answered and refused the handshake, so it can never be
adopted; preserving it repeated the same failed adoption on every launch,
forever. And the veto read the process table, not the socket, so it could
fire on a daemon whose endpoint was already gone — adoption then threw,
init aborted, and the app was left with no spawner and no working Restart.
Gate on both: only preserve what could still be reached.

Also: releasing the launcher's temporary lease after the permanent lease
failed reopened the adoption gap that ordering exists to close, and the
tolerance added to discoverDaemonSessions was dead code — nothing on that
path rejects.

Co-authored-by: Orca <help@stably.ai>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/main/daemon/daemon-live-pty-evidence.ts (1)

15-19: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Reduce the long implementation comments.

Keep the policy and platform constraints. Remove the detailed examples and repeated rationale. State each reason in one or two lines.

As per coding guidelines, “Comments must be concise, non-obvious, and brief—prefer one line; do not explain obvious behavior or walk through code.”

Also applies to: 62-71, 135-142

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 928b6dcc-fa3f-404d-a04c-03667a244e29

📥 Commits

Reviewing files that changed from the base of the PR and between 12a1366 and 8b46147.

📒 Files selected for processing (3)
  • src/main/daemon/daemon-init.ts
  • src/main/daemon/daemon-live-pty-evidence.test.ts
  • src/main/daemon/daemon-live-pty-evidence.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/main/daemon/daemon-live-pty-evidence.test.ts
  • src/main/daemon/daemon-init.ts

nwparker and others added 4 commits August 11, 2026 20:26
Three review rounds each found a new failure state in the previous shape,
which was the design telling us something. The safety rule — never destroy
running work — was replicated across the launcher's branches instead of
being decided once, and the last round added it to one more branch behind a
boolean. Policy had been put inside a mechanism: killStaleDaemon grew an
input flag to disable its new veto, an output back-channel to report it, and
a caller-side re-derivation of the classification the flag had lost. One
structural error, one symptom per layer it crossed.

Name the question instead. resolveDaemonOccupancy answers occupied | empty |
unknown, asking the daemon first (authoritative both ways) and falling back
to the process table only to RAISE the answer to occupied. OS evidence can
prove work exists; it can never prove absence, so it never licenses a kill.

The launcher now decides before it destroys anything, so killStaleDaemon
goes back to being only "make this pid go away" — no options, nothing for a
future caller to forget to disable, and Manage Sessions -> Restart cannot be
vetoed because there is no veto left to hit.

Holding is a real outcome now. A daemon that owns live terminals but cannot
answer a handshake gets mode 'held': no adoption attempt, no lease, no fork
beside it. That deletes the lease-free-handle fallback, the try/catch around
preserve, and the tolerated-lease branch in init that existed only because
the correct outcome had no representation.

Two defects this removes outright:

- The endpoint check used a local boolean probe that returns false on
  timeout, so under load — and unconditionally on Windows named pipes, where
  a busy server answers ERROR_PIPE_BUSY — the guard disabled itself in
  exactly the conditions it was written for. Use the canonical three-valued
  probe, whose own docs say absence of proof is not proof of death.
- A daemon that answered and refused the handshake was killed with its
  agents. It is now held like any other occupied daemon.

Also folds the replacement verdict onto pendingReplacement, retiring a pair
of mutable launcher locals whose only job was moving one warning past the
kill.

Co-authored-by: Orca <help@stably.ai>
The module's contract is that 'unknown' is where every unanswerable question
lands, but a throwing dependency escaped instead — routing a failed
observation into the launch path rather than onto the safe residual. Latent
today because both real implementations swallow their own failures, which is
exactly the kind of thing that stops being true during a refactor.

Co-authored-by: Orca <help@stably.ai>
Round-four review found the evidence proving the wrong thing. The filter
excluded the daemon's plain subprocesses on the grounds that only a PTY child
is a session leader — but the daemon opens PTYs for its own health probe and
conpty warmup, and forkpty makes those session leaders too. The comment's own
premise refuted its exclusion list. A daemon hosting zero user terminals could
be held on the strength of its stuck probe child, and since the held daemon
also had no sessions, dropping our authenticated pair let it retire and take
the very state we were protecting. Exclude them by exact command, on both
platforms.

Two more from the same review:

- pty-spawn-unhealthy is only reachable after a successful hello, so that
  daemon is adoptable. It was routed to 'held' — which never adopts — purely
  because the count had come from the process table. Check it first; hold now
  requires an unreachable daemon.
- The grace loop rescanned the process table every pass, though it is waiting
  for IPC and the table cannot change its answer in five seconds. Ask the
  daemon during the wait and read the table once, after. raiseOccupancy-
  WithProcessEvidence makes that split explicit, and can only ever raise.

Bound the wait by wall clock too: the retry count alone never bounded it, and
startup fails open at 60s by abandoning the daemon provider outright, which
would trade a wedged daemon for no daemon and a Restart that throws.

Co-authored-by: Orca <help@stably.ai>
…t refused us

Found while reviewing why a mutation looked equivalent. Gating the hold on
health === 'unreachable' left 'rejected' — a daemon that answered and refused
the handshake — falling through to preserveDaemon(), whose adoption opens the
very hello it just refused. That throws, and the throw costs the app its
daemon and its Restart remedy. Killing it instead is no better: it can still
be hosting running agents.

Neither of those daemons can complete a handshake, so neither may be adopted,
and both must be held. Gate on that rather than on one of its two causes.

Adds regression tests for the round-four fixes: the self-spawned probe
exclusion is exact-match on both platforms, an adoptable pty-spawn-unhealthy
daemon is never routed to a mode that cannot adopt, evidence can only raise a
verdict, and the grace budget stays under the startup fail-open cap.

Co-authored-by: Orca <help@stably.ai>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/main/daemon/daemon-init.test.ts (1)

97-100: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Reset the two new mocks in importFresh().

importFresh() resets every other shared mock at Lines 459-514, but not inspectDaemonPtyOwnershipMock or readVerifiedDaemonPidMock. The new tests restore both in their finally blocks, so the suite passes today. A future test that sets either mock and returns early would leak owns-live-ptys or a verified pid into later tests, which would silently convert a replacement case into a preserve case.

♻️ Proposed reset in `importFresh()`
   getProcessStartedAtMsMock.mockReset()
   getProcessStartedAtMsMock.mockReturnValue(1_000_000)
+  inspectDaemonPtyOwnershipMock.mockReset()
+  inspectDaemonPtyOwnershipMock.mockResolvedValue('unknown')
+  readVerifiedDaemonPidMock.mockReset()
+  readVerifiedDaemonPidMock.mockResolvedValue(null)

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 35763b87-f41a-498c-8c31-c43524117efd

📥 Commits

Reviewing files that changed from the base of the PR and between 8b46147 and cb0f3e4.

📒 Files selected for processing (10)
  • config/scripts/daemon-replacement-live-agent-pty-preservation-repro.mjs
  • src/main/daemon/daemon-health.test.ts
  • src/main/daemon/daemon-health.ts
  • src/main/daemon/daemon-init.test.ts
  • src/main/daemon/daemon-init.ts
  • src/main/daemon/daemon-live-pty-evidence.test.ts
  • src/main/daemon/daemon-live-pty-evidence.ts
  • src/main/daemon/daemon-occupancy.test.ts
  • src/main/daemon/daemon-occupancy.ts
  • src/main/daemon/daemon-spawner.ts
💤 Files with no reviewable changes (1)
  • src/main/daemon/daemon-health.test.ts
🚧 Files skipped from review as they are similar to previous changes (3)
  • src/main/daemon/daemon-health.ts
  • src/main/daemon/daemon-live-pty-evidence.ts
  • config/scripts/daemon-replacement-live-agent-pty-preservation-repro.mjs

Comment thread src/main/daemon/daemon-init.test.ts
…one place

Round five found the same question — can this daemon complete a hello right
now? — answered in three places with three different conclusions, because
'held' had been added as a fifth branch rather than as the classification the
other branches route through. Two of those answers were wrong, and both ended
in no daemon at all, which is the outcome 'held' exists to prevent.

- A daemon whose adoption hello had just failed was handed back tagged
  'degraded-new-pty-fallback'. Init skips the lease only for 'held', so it
  reopened the same connection, threw, and aborted startup — leaving the
  agents alive but unreachable and Manage Sessions -> Restart throwing.
- The pty-spawn-unhealthy arm ran first and claimed a successful hello proved
  adoptability, but that reading is from before the grace window. A daemon
  that answered at t=0 and went silent through thirty seconds of retries took
  that arm and threw the same way. Ask whether it is answering now, first.

The budget was a comment with a Date.now() beside it: one occupancy probe
could cost 50s, because the client's default is a 5s hello per connection
step plus a 30s request timeout. Bound the probe explicitly, start the clock
before the first one, and size the window so the whole path — health check,
pid verification, loop overshoot and process-table read — fits under the
startup fail-open with room to spare.

Also folds the four sibling branches onto the same occupancy resolution.
getAliveDaemonSessionCount was byte-identical to countLiveSessionsOverIpc, so
one concept had two implementations and only one of them had been fixed.

The Windows probe exclusions could never match: the warmup spawns COMSPEC, an
absolute path, against an exact-equality test on 'cmd.exe /c exit'. Compare
the program by basename and keep the argv tail exact.

Co-authored-by: Orca <help@stably.ai>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 257cef3c-863d-4881-a75f-4878e6fc31d6

📥 Commits

Reviewing files that changed from the base of the PR and between cb0f3e4 and f5abf2a.

📒 Files selected for processing (5)
  • config/scripts/daemon-replacement-live-agent-pty-preservation-repro.mjs
  • src/main/daemon/daemon-init.test.ts
  • src/main/daemon/daemon-init.ts
  • src/main/daemon/daemon-live-pty-evidence.ts
  • src/main/daemon/daemon-occupancy.ts
🚧 Files skipped from review as they are similar to previous changes (4)
  • src/main/daemon/daemon-live-pty-evidence.ts
  • src/main/daemon/daemon-occupancy.ts
  • config/scripts/daemon-replacement-live-agent-pty-preservation-repro.mjs
  • src/main/daemon/daemon-init.ts

Comment thread src/main/daemon/daemon-init.test.ts
nwparker and others added 2 commits August 11, 2026 21:38
…ot own

Closing a held daemon's pane reported success while the agent kept running.
Unrouted ids resolve to the in-process fallback, whose shutdown returns
silently for an id it has never heard of and whose write and resize are
no-ops — and while a daemon is held nothing ever enumerates its sessions, so
every one of them is unrouted. The pane vanished, the orphan outlived the
app, and typing into a stuck terminal disappeared without a word.

Only attach was fenced against that route. Extend the same rule to the
operations that change or feed a session: route to the fallback only when it
genuinely owns the pty, and otherwise say the session cannot be reached.

The error type is load-bearing. pty:kill treats "Session not found" as proof
the pty is already gone and synthesizes an exit, so reusing that error would
have reproduced the lie one layer down. TerminalSessionOwnerUnverifiedError
means "still there, we cannot reach its host", which is reported as a failed
close and keeps ownership for a retry.

Co-authored-by: Orca <help@stably.ai>
…that never had it

Covers the held-daemon routing fence, including the coupling that is
invisible from the routing file: the thrown error must not match pty:kill's
already-gone predicate, or the close is swallowed into a synthesized exit and
the orphan is hidden again. A rename would otherwise reintroduce the bug
silently.

Co-authored-by: Orca <help@stably.ai>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/main/daemon/degraded-daemon-session-routing.ts (1)

29-40: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Shorten the new comments.

Keep the ownership rationale, but reduce each comment to one concise sentence.

  • src/main/daemon/degraded-daemon-session-routing.ts#L29-L40: Replace the multi-paragraph operation-routing explanation with a brief ownership-safety rationale.
  • src/main/daemon/degraded-daemon-pty-provider.test.ts#L691-L692: Reduce the held-daemon fixture explanation to one sentence.
  • src/main/daemon/degraded-daemon-pty-provider.test.ts#L711-L712: Reduce the shutdown rationale to one sentence.
  • src/main/daemon/degraded-daemon-pty-provider.test.ts#L723-L726: Move detailed pty:kill behavior to the test name or use one short statement.

As per coding guidelines: “Comments must be concise, non-obvious, and brief—prefer one line.”

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ec5a9074-0e5d-48c0-9625-6eb3198fe4e1

📥 Commits

Reviewing files that changed from the base of the PR and between f5abf2a and 246da42.

📒 Files selected for processing (3)
  • src/main/daemon/degraded-daemon-pty-provider.test.ts
  • src/main/daemon/degraded-daemon-pty-provider.ts
  • src/main/daemon/degraded-daemon-session-routing.ts

nwparker and others added 12 commits August 11, 2026 22:04
…escribes

Round-six follow-ups, none destructive.

The process-table read had a deadline on Windows but not on POSIX, where the
shared reader's ps timeout does not cover queueing behind an in-flight scan —
so the one step that runs after the grace window could still outlast it.

The pid handed to that read was verified before the grace window, which is
long enough for the daemon to die and its pid to be recycled onto a shell
with children. Verify it where it is used instead, and only when there is
still something to raise: the common case now skips the identity probe
altogether, which also takes a few seconds off the worst-case launch.

Two renderer call sites killed PTYs without handling rejection. That was
harmless while an unreachable session was answered by a silent no-op; now
that it honestly rejects, pane teardown and repo removal would log an
unhandled rejection every time — exactly when the daemon is already sick.

Deliberately not taken from that review: giving the endpoint-occupied catch
the same held fallback as the failed-health path. That path arrives with
occupancy unknown or empty, so holding there would swallow a real launch
failure to protect nothing.

Splits the repro script, which had grown past the line limit, into the
sequence it proves and the two things it proves it with: process-table
inspection, and the static assertions on the launcher's hold decision.

Co-authored-by: Orca <help@stably.ai>
…ne term

The previous assertion compared the grace window to the fail-open cap, which
passed while the real path ran to roughly twice the cap — a single probe cost
50s against a 5s assumption, and the terms on either side of the loop were
never counted at all.

Sum the declared budgets instead: health check, grace window, the one probe
that always runs past a ceiling tested at loop entry, and the evidence read
on both platforms. Raising any of them now has to face this, and the spare
time the kill ladder and fork still need afterwards is stated rather than
assumed.

Lives outside the launcher's own spec because that file mocks daemon-health,
which would shadow the constants being held to account.

Co-authored-by: Orca <help@stably.ai>
Raised by a colleague's handoff on the same-day reports. macOS wraps every
terminal in /usr/bin/login for TCC attribution, and #13764 shows the wrapper
can outlive the shell it wrapped — leaving a session leader that hosts
nothing. One affected host had accumulated enough of them to reach swap
pressure.

That is exactly the evidence this change treats as proof of live work, so a
daemon whose sessions had all ended would have been held indefinitely on the
strength of the corpses, on precisely the hosts where the problem is worst.
Same class as the daemon's own probe PTYs: a session leader is necessary
evidence, not sufficient. A wrapper still doing its job has the shell it
exec'd beneath it.

Co-authored-by: Orca <help@stably.ai>
…cannot silently rot

The ownership evidence discounts the PTYs the daemon opens for itself, and that
list is only safe while it is complete — a self-spawned PTY nobody excluded
reads as user work and holds a daemon that owns nothing. The list grew one
reviewer at a time, which is the wrong mechanism for a correctness invariant.

Pin the input rather than the list. The daemon has exactly three PTY spawn
sites: the user's terminal, the spawn health probe, and the Windows conpty
warmup. A fourth now fails this test until someone decides which side it
belongs on.

Co-authored-by: Orca <help@stably.ai>
…rotect

Readiness review, section 04. Every agent pane already drives the shared
process-table reader on its own cadence, so the uncached read queues behind
them — and the host with the most agents to lose is the one likeliest to blow
the deadline on queueing alone. Both attempts return unknown and the daemon is
killed anyway, which is the original bug wearing the fix as a costume.

Fall back to the TTL-cached table, which on that host is always warm for
exactly the reason the uncached read is always queued. A table a few hundred
milliseconds old still answers whether this daemon has children, and the
failure directions are not symmetric: over-holding costs one degraded launch
that self-heals, under-counting ends running agents.

The same review found the launch budget still overran the 60s startup
fail-open — by ~7s on Windows — and that the test guarding it under-counted
the path it was written to bound, for the second time. It omitted the identity
probe before the evidence read and the endpoint check that ends the grace
loop. Both are now summed, the headroom requirement covers the kill ladder and
fork that follow a replace verdict, and the grace window and Windows probe
deadline are sized to fit.

Not taken from that review: reusing the verified pid inside killStaleDaemon to
drop the duplicate probe. That second verification is what fences the signal to
this incarnation, and a seconds-old result is exactly the pid-reuse hazard it
exists to prevent.

Co-authored-by: Orca <help@stably.ai>
Readiness review, sections 02/03/05/06 — no P0 or P1 in any of them. These are
the P2s worth taking.

The important one: on macOS a terminal contributes exactly one session leader,
the login wrapper, because the shell it forks is in the same session and shows
S+ rather than Ss. I had assumed the shell counted too. It does not — so a
wrapper that looks childless in a single snapshot makes its whole terminal
invisible, and that snapshot cannot tell a wrapper whose shell has gone from
one whose shell has not yet appeared. Emptiness is the answer that authorizes a
kill, so it now costs a second read; 'owns-live-ptys' still needs none. The
fixtures said Ss where a real shell says S+, which is why the tests never
noticed.

Also from that review: a fabricated row was cast to ProcessTableRow to reuse a
command-only predicate, which is sound only while that predicate reads nothing
else — narrowed to Pick<'command'> so the compiler keeps it honest. The
pty:signal listener relied on its provider staying async to convert a routing
refusal into a rejection; it is an ipcMain.on listener with nothing above it, so
it now catches synchronously too. And the repro's teardown signalled remembered
pids a minute after phase 1 waited for them to die — re-verify by tag first,
since signalling a recycled pid is the mistake the script exists to study.

Co-authored-by: Orca <help@stably.ai>
…etter

The readiness review found the Windows branch reading a wedged daemon's
orphaned conpty hosts as live terminals. ClosePseudoConsole only runs on the
daemon's own JS thread, so a daemon too wedged to answer is also too wedged to
reap them, and they accumulate exactly when this code runs. A wedged, empty
Windows daemon would then be held forever — #8689 re-opened, and a regression
from main rather than a missing protection.

The tempting fix is another exclusion. That would be the sixth revision to what
counts as a live PTY, each one added because a reviewer found something that
looks like a session and is not, and each one trading safety for availability
in a fix whose entire purpose is the opposite trade. The list is the problem.

POSIX has a real signal: forkpty makes a hosted terminal a session leader, which
nothing the daemon forks for itself ever is. Windows has no equivalent, so its
branch could only ever count descendants and subtract guesses. Delete it and
answer 'unknown' — Windows keeps exactly the behaviour it has on main, and the
protection is claimed only where it can be justified.

Also stops a blind confirming read from upgrading an unconfirmed emptiness into
a verdict. Emptiness is what authorizes a kill; a read that saw nothing
corroborates nothing.

Co-authored-by: Orca <help@stably.ai>
Behaviour-preserving. Windows now abstains once at the entry point rather than
twice inside a retry loop that had nothing to retry, which also retires the
platform check further down that could no longer be false. The self-spawn
matcher kept backslash splitting and .exe stripping for a branch that no longer
exists, and the launcher's grace loop repeated its own IPC call and stacked two
explanations above the wrong statement.

Co-authored-by: Orca <help@stably.ai>
Round seven found the one thing this PR must never do: kill a session that main
would have kept.

Main's grace loop probed with a non-shared 5s connect budget, so a wedged
daemon got roughly a minute to come back. Bounding the probes and adding a
wall clock cut that to about twelve seconds — a good trade on POSIX, where a
daemon that outlasts the window is still protected by process-table evidence,
and a bad one on Windows, which has no such evidence and now has nothing else.
A Windows daemon wedged for half a minute while hosting agents was adopted by
main and is killed by this branch. The fail-open cannot rescue it either:
ensureRunning() is not abortable, so the launcher runs to completion.

Size the window per platform instead, against what each actually spends:
Windows pays no evidence read and no identity probe to feed one, so it can
afford far more grace, and grace is worth more where it is the only thing
there. Both numbers come from the budget test rather than taste.

Three more from the same review:

- The evidence read applied its deadline twice, once to the fresh table and
  again to the cached fallback, so an attempt could cost double what the launch
  budget was told. Share one deadline across both.
- Two tests described protection the code no longer delivers: one asserted ~60s
  of grace the wall clock had already retired, the other passed only because its
  mocked probes are free and would fail against real ones. Say what the code
  actually promises, and freeze the clock where the point is retry depth.
- The self-spawned PTY inventory promised more than it inspects. It sees direct
  node-pty calls in one directory; the macOS login-session probe reaches a PTY
  through expect(1) and is caught by the stranded-wrapper filter instead. Scope
  the claim, since that indirection is the shape the next escape will take.

Co-authored-by: Orca <help@stably.ai>
…a sum

Round eight found the fourth term missing from the hand-written budget — the
launcher's own adoption connect, which runs before the health check on the
non-shared five-second path. The three before it were an identity probe, an
endpoint probe, and an evidence deadline applied twice. Every one of them
passed the test meant to catch exactly that, because the test could only check
the terms someone had remembered to add.

So stop summing. The classification now runs against a deadline and stops when
it expires, and the test asserts only that the deadline leaves room for the
kill ladder and the fork that follow it. A budget that has to be remembered is
a budget that will be wrong; this one cannot be, because nothing has to be
counted.

That also retires the platform-split grace window, which existed to hand
Windows more of a sum nobody could total correctly.

The same review found the regression it was compensating for was never the
window. main gave each probe up to fifty seconds — five per connection step,
thirty for the request — where this branch gave eight for both together. A
daemon whose handshake needs more than four seconds therefore answered none of
the probes, however many it got, and on Windows nothing else can speak for it.
Splitting the two budgets fixes the case the window never could: connecting
stays tight, because a daemon that cannot handshake is wedged and worth
re-asking cheaply, while a daemon that did handshake is demonstrably alive and
its count is worth waiting for.

Also stops a Date.now spy leaking out of a failed test and freezing the clock
for the rest of the file.

Co-authored-by: Orca <help@stably.ai>
… names

The clock introduced in the previous commit gated the probes but not the two
steps after them. The identity re-check and the process-table read ran on their
own deadlines, outside the ceiling, so the launcher could still spend its whole
budget on probes and then take another ten seconds — the same overrun the sum
used to produce, arrived at from the other end.

Hold that time back from every probe instead. A probe is only started when the
clock can still fund a handshake after the reserve, and its budget is what
remains minus the reserve, so no probe can eat it however long the daemon takes
to answer. Worst case is now the ceiling by construction rather than by
addition.

The reserve has a test asserting it is large enough for what it covers, which
failed on its first run and caught that ten seconds was not.

Co-authored-by: Orca <help@stably.ai>
Round nine found the re-verification was stricter than the check that triaged
the daemon onto this path. The launcher gets here because a three-second health
check — one socket, one hello — timed out. It then re-asked with two sockets
and two hellos inside four shared seconds, and repeated that identical question
up to twelve times. A daemon that consistently needs five seconds fails every
one of them, so the retries could only ever agree with the check that sent it
here. main re-asked with five seconds per connection step and thirty for the
answer, and kept the sessions this branch destroyed.

Retries and patience solve different problems. Keep the cheap probes, which
catch a daemon that recovers on its own, then spend what is left of the clock
on one tolerant ask — the only question that can disagree with the triage. It
is skipped when the endpoint is provably gone, since a cold start arrives here
too and has nothing to wait for.

Two more from the same review. The clock claimed to cover the launcher's own
adoption connect and started after it, so the fourth term that went missing
from the sum was still uncounted; it now starts above that connect and bounds
it. And Windows was holding back twelve seconds for an identity check and a
process-table read it never performs — the reserve is zero where the steps it
reserves for do not run.

Retuned the ceiling to leave the kill ladder and fork real margin rather than
half a second, with the packaged-Windows host copy named as what the margin is
for.

Co-authored-by: Orca <help@stably.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant