E2E docs follow SDD; a Resume at the instant of finish is no longer refused as busy

.changeset/resume-at-finish-not-refused.md

@@ -0,0 +1,5 @@
+---
+'@gemstack/the-framework': patch
+---
+
+Resuming a session the instant it finishes is no longer spuriously refused as "already active". A run's row flips `done` the moment the child writes its ending, but the child takes a beat more to actually exit — and a Resume landing in that gap found the run's slot still holding a live pid and was turned away by the busy guard, though the session was over by its own account (the E2E settings story caught this on a slow CI runner). The start path now waits out a finished leg's exit and the retirement queued behind it — the settle chain is parked per run slot so the continuation can await it — and only then judges the guard, so a refusal is reserved for sessions that are genuinely still running and the checkout reuse always reads a settled archive.

packages/the-framework/spec.md

@@ -49,7 +49,7 @@ flowchart TD
 
 **Settle and handoff.** Settling is strictly ordered: a final quality turn (which queues the quality presets as backlog entries rather than running them, and folds new learnings into the project docs) → the git handoff → close and archive the session history. The handoff runs only on the success path (a user-stopped or failed session skips it) and first decides whether the session is *empty* — no commits, or only bookkeeping files changed. Empty sessions are never published; otherwise pending work is committed, the branch pushed, and a PR opened. Both halves default on — that is the zero-config promise: a session left alone publishes itself. They form a ladder, not a pair: pushing is the rung under the PR, so turning push off publishes nothing, and "PR without push" is not a state a session can be asked for.
 
-**Teardown and retention.** A session that finished cleanly has its worktree removed; one that failed or was stopped keeps its checkout, because that is exactly when you want to inspect the half-finished tree. The branch and the archived session history always survive the worktree. A session that died on a transient error (connection drop, rate limit) is retried in the same worktree before being declared failed, and a finished session can be reopened later — its history restored so it continues as the same conversation rather than starting empty.
+**Teardown and retention.** A session that finished cleanly has its worktree removed; one that failed or was stopped keeps its checkout, because that is exactly when you want to inspect the half-finished tree. The branch and the archived session history always survive the worktree. A session that died on a transient error (connection drop, rate limit) is retried in the same worktree before being declared failed, and a finished session can be reopened later — its history restored so it continues as the same conversation rather than starting empty. Acting on a session the instant it finishes is safe: everything that touches that session's checkout — the teardown itself, publishing, reopening, deleting — takes its turn rather than racing, so a click that lands mid-teardown waits a beat instead of failing.
 
 ### Autonomy
 

packages/the-framework/src/daemon-runtime.test.ts

@@ -0,0 +1,75 @@
+import { strict as assert } from 'node:assert'
+import { test } from 'node:test'
+import { waitOutFinishedLeg } from './daemon-runtime.js'
+
+const sleep = (ms: number): Promise<void> => new Promise(resolve => setTimeout(resolve, ms))
+
+const emptySlots = (): { starting: Set<string>; activeRuns: Map<string, number>; retiring: Map<string, Promise<void>> } => ({
+  starting: new Set<string>(),
+  activeRuns: new Map<string, number>(),
+  retiring: new Map<string, Promise<void>>(),
+})
+
+// The seam behind #1529: a Resume fired the instant a run flips `done` used to reach the busy
+// guard while the child that wrote that ending was still mid-exit, and the guard refused a
+// session that was over by its own account. These pin the wait's decision table; the settings
+// story exercises the full daemon path.
+
+test('a free slot returns immediately, without even asking whether the leg ended', async () => {
+  let asked = 0
+  await waitOutFinishedLeg(
+    'p::r1',
+    emptySlots(),
+    async () => {
+      asked += 1
+      return true
+    },
+    5_000,
+  )
+  assert.equal(asked, 0)
+})
+
+test('a leg still calling itself running is a real collision: no wait, the refusal stands', async () => {
+  const slots = emptySlots()
+  slots.activeRuns.set('p::r1', process.pid)
+  const before = Date.now()
+  await waitOutFinishedLeg('p::r1', slots, async () => false, 5_000)
+  assert.ok(Date.now() - before < 1_000, 'must not sit out the grace period')
+  assert.ok(slots.activeRuns.has('p::r1'), 'the slot is left for the busy guard to judge')
+})
+
+test('a finished leg is waited out: the wait ends once the exit clears the slot', async () => {
+  const slots = emptySlots()
+  slots.activeRuns.set('p::r1', process.pid)
+  setTimeout(() => slots.activeRuns.delete('p::r1'), 60)
+  await waitOutFinishedLeg('p::r1', slots, async () => true, 5_000)
+  assert.equal(slots.activeRuns.has('p::r1'), false)
+})
+
+test('the wait is bounded: a finished leg whose process never exits falls back to the guard', async () => {
+  const slots = emptySlots()
+  slots.activeRuns.set('p::r1', process.pid)
+  const before = Date.now()
+  await waitOutFinishedLeg('p::r1', slots, async () => true, 120)
+  assert.ok(Date.now() - before >= 120, 'the grace period was sat out')
+  assert.ok(slots.activeRuns.has('p::r1'), 'the still-occupied slot reaches the busy guard')
+})
+
+test('a retirement already in flight is awaited even after the exit cleared the slot', async () => {
+  const slots = emptySlots()
+  let retired = false
+  slots.retiring.set(
+    'p::r1',
+    sleep(60).then(() => {
+      retired = true
+    }),
+  )
+  await waitOutFinishedLeg('p::r1', slots, async () => true, 5_000)
+  assert.equal(retired, true)
+})
+
+test('a retirement that failed does not fail the continuation waiting on it', async () => {
+  const slots = emptySlots()
+  slots.retiring.set('p::r1', Promise.reject(new Error('archive on a full disk')))
+  await waitOutFinishedLeg('p::r1', slots, async () => true, 5_000)
+})

packages/the-framework/src/daemon-runtime.ts

@@ -14,6 +14,7 @@ import {
   attachWorktree,
   worktreePath,
   listRuns,
+  findRun,
   archivedRunPaths,
   commitPendingWork,
   currentBranch,
@@ -238,6 +239,9 @@ export const MAX_TRANSIENT_RETRIES = 2
 /** The pause before a retry (#1281): long enough for a dropped connection to be worth re-trying. */
 const TRANSIENT_RETRY_DELAY_MS = 15_000
 
+/** How long a continuation waits for its finished previous leg to exit and retire (#1529). */
+const FINISHED_LEG_EXIT_GRACE_MS = 15_000
+
 /** What the continued session is told (#1281), in the #923 resume prompt's shape. */
 const RETRY_PROMPT =
   'This session died to a transient connection error, not because anyone asked it to stop. Look at what you had already done, then carry on from there and finish the work.'
@@ -344,6 +348,30 @@ async function waitForExit(pid: number, timeoutMs: number): Promise<boolean> {
   return !isPidAlive(pid)
 }
 
+/**
+ * Wait out the previous leg of the run a continuation is aimed at (#1529). A Resume clicked the
+ * instant a run's row flips `done` can land while the child that wrote that ending is still
+ * mid-exit: the run's slot then still holds a live pid, and the busy guard read "already active"
+ * off a session that is over by its own account — a spurious refusal the E2E settings story
+ * caught on a slow runner. A finished child's exit is imminent and its retirement is queued
+ * right behind it (see `retiring` in {@link createProjectRuntime}), so wait for both, bounded by
+ * `graceMs`, and let the reuse read a settled archive. A leg still calling itself `running` is a
+ * genuine collision: not waited on, so the guard's refusal stands.
+ */
+export async function waitOutFinishedLeg(
+  key: string,
+  slots: { starting: Set<string>; activeRuns: Map<string, number>; retiring: Map<string, Promise<void>> },
+  legHasEnded: () => Promise<boolean>,
+  graceMs: number,
+): Promise<void> {
+  const occupied = (): boolean => slots.starting.has(key) || slots.activeRuns.has(key)
+  if (!occupied() && !slots.retiring.has(key)) return
+  if (!(await legHasEnded())) return
+  const deadline = Date.now() + graceMs
+  while (occupied() && Date.now() < deadline) await delay(25)
+  await slots.retiring.get(key)?.catch(() => {})
+}
+
 /** Inputs to {@link createProjectRuntime}. */
 export interface ProjectRuntimeOptions {
   /** The daemon's home workspace; a run/preview with no project id targets it. */
@@ -402,6 +430,15 @@ export function createProjectRuntime({ cwd, env, binPath, retryDelayMs, agentPre
   // Live run pids, keyed per run rather than per project (#736) — see onStart for the key.
   const activeRuns = new Map<string, number>()
   const starting = new Set<string>() // reserved keys mid-spawn, to close the async gap
+  // A finished leg's exit → retirement chain, parked per run slot so a continuation that raced
+  // the exit (#1529) can await the retirement instead of reusing a checkout mid-removal.
+  const retiring = new Map<string, Promise<void>>()
+  const parkRetirement = (key: string, retired: Promise<void>): void => {
+    retiring.set(key, retired)
+    void retired.finally(() => {
+      if (retiring.get(key) === retired) retiring.delete(key)
+    })
+  }
   // Runs this daemon is relaying to/from a connected device (#1067): the local half of a remote run.
   const relayedRuns = new RelayedRuns()
   // The relayed-run lookup the dashboard's read RPCs consult (#1067 slice 2): is this runId remote, and
@@ -693,7 +730,12 @@ export function createProjectRuntime({ cwd, env, binPath, retryDelayMs, agentPre
       if (!movedRunId) return
       // The moved meta is normally already there, so the marker no-ops; it only writes when the
       // copy was torn AND the resumed child died at boot (#1261) — the same hang either way.
-      void markFailedStart(checkout, movedRunId, '', detail).finally(() => void tearDownWorktree(projectCwd, checkout, movedRunId))
+      parkRetirement(
+        key,
+        markFailedStart(checkout, movedRunId, '', detail)
+          .catch(() => {})
+          .then(() => tearDownWorktree(projectCwd, checkout, movedRunId)),
+      )
     }
     continued.once('error', err => settle(`its process could not be spawned (${errorMessage(err)})`))
     continued.once('exit', (code, signal) => settle(exitDetail(code, signal)))
@@ -855,6 +897,23 @@ export function createProjectRuntime({ cwd, env, binPath, retryDelayMs, agentPre
     // its options client-side and sends them whole.
     if (options.continueRunId) {
       options = { ...(await resolveProjectRunOptions(projectKey, env)), ...options }
+      // A Resume fired the instant its run flips `done` can also land while the child that wrote
+      // that ending is still mid-exit (#1529): the slot then still holds a live pid, and the busy
+      // guard below refused a continuation of a session that is over by its own account. Wait the
+      // exit and its queued retirement out, so the guard judges only real collisions and the
+      // checkout reuse reads a settled archive.
+      const { continueRunId } = options
+      await waitOutFinishedLeg(
+        scopedKey(projectKey, continueRunId),
+        { starting, activeRuns, retiring },
+        async () => {
+          // The composed read (live meta wins over archive): the leg just wrote `done` into its
+          // worktree and teardown has not archived it yet, so the archive-only list cannot see it.
+          const meta = continueRunId ? await findRun(projectCwd, continueRunId).catch(() => undefined) : undefined
+          return meta !== undefined && meta.status !== 'running'
+        },
+        FINISHED_LEG_EXIT_GRACE_MS,
+      )
     }
 
     // A run must not spend a branch and a worktree on an agent that can never start (#1326).
@@ -918,10 +977,13 @@ export function createProjectRuntime({ cwd, env, binPath, retryDelayMs, agentPre
         // The failed marker lands before the teardown reads the meta (#1261), so a boot death is
         // archived as `failed` and the worktree is then kept for inspection, not removed. After
         // teardown the archive is readable, which is when a transient death earns a retry (#1281).
-        void markFailedStart(checkout, runId, prompt, detail).finally(() =>
-          void tearDownWorktree(projectCwd, checkout, runId).finally(() =>
-            void retryTransientDeath(projectCwd, targetProjectId, runId, options),
-          ),
+        parkRetirement(
+          key,
+          markFailedStart(checkout, runId, prompt, detail)
+            .catch(() => {})
+            .then(() => tearDownWorktree(projectCwd, checkout, runId))
+            .then(() => retryTransientDeath(projectCwd, targetProjectId, runId, options))
+            .catch(() => {}),
         )
       }
       child.once('error', err => settle(`its process could not be spawned (${errorMessage(err)})`))

packages/the-framework/src/dashboard-rpc/spec.md

@@ -2,7 +2,7 @@ The browser's call surface into the daemon: every dashboard read and write arriv
 
 ## TLDR
 
-- Reads are thin projections of the daemon's read models; live session events stream over one channel per session, tailing that session's own log.
+- Reads are thin projections of the daemon's read models; live session events stream over one channel per session, tailing that session's own log. A watcher never misses a session's ending: when the log is archived out from under a live stream at teardown, the stream follows it into the archive and delivers exactly what it had not yet shown — once.
 - Writes are commands: stop, answer a choice, send a message, arm the handoff, push, open PR, merge, start a session, queue a ticket. Each becomes an entry in the target session's control file, which the session tails — there is no direct channel into the running process.
 - Two routing decisions live here and nowhere else: which checkout a session-scoped call resolves to (the session's worktree vs. the project root), and whether a call is local or belongs to a relayed session on a remote device — in which case it is forwarded, against a deliberate allowlist on the device side.
 

packages/the-framework/src/e2e/README.md

@@ -0,0 +1,21 @@
+# Backend E2E story tests
+
+Each test walks one dashboard user story through the real daemon runtime — real registry, real git repos, real spawned run processes — with the deterministic `--fake` driver standing in for the coding agent, so the whole flow runs offline in seconds and no browser is involved. They run as ordinary tests under `scripts/run-tests.mjs`, alongside the unit suites.
+
+## What "end-to-end" means here
+
+- **The entry points are the dashboard's own RPCs.** Stories call the same telefunctions the browser invokes over `/_telefunc` (`sendStart`, `sendChoice`, `onRuns`, `onQueue`, …), with the request context wired exactly the way `runDaemon` wires it: the runtime's `onStart`/`onAddProject`/`preview` closures, the registry-backed preferences store, and stub quota/auto-PM reporters where the daemon would hold live pollers.
+- **Runs are real processes.** `createProjectRuntime` spawns each run detached, exactly as the daemon does; the harness's `binPath` points at `fake-agent-bin.ts` (compiled beside it), which forwards to the real CLI with `--fake` appended. Everything between the Start click and the archived run row — worktree allocation, the run store, `events.jsonl`, the control watcher, gates, teardown, retention — is the production code path; only the agent turn is scripted.
+- **The Telefunc transport hop is out of scope.** Telefunc `Channel`s only pump over a real wire, so live-stream assertions ride `tailRunEvents` — the same tailer `onEvents` wraps. The mount, CSRF/rebinding guards, and channel plumbing have their own tests (`dashboard/server.test.ts`, `dashboard-rpc/stream-channel.test.ts`).
+
+## Isolation
+
+`harness.ts` points `$XDG_CONFIG_HOME` at a fresh temp dir per test process, so a story file's registry (projects, preferences, daemon state) can never see — or be seen by — the sibling test files running concurrently. Every world lives in temp dirs and `close()` kills whatever runs it spawned.
+
+## Scripting the fake agent
+
+`FRAMEWORK_FAKE_AWAIT=choices|multiselect|confirmation` (set before a Start; spawned children inherit the env) makes the fake agent's first turn end on that gate, which is how stories park a run on a question deterministically. Without it the fake agent answers every prompt with one scripted build turn and the run ends on its own.
+
+## The finished-session seam
+
+The publish and resume stories fire *at the moment* a run's meta flips `done` — inside what used to be a race window — and then assert teardown still retires the worktree. They are the regression tests for the per-run checkout lock (`run-locks.ts`) and the archive-following event tail (`events-tail.ts`). `waitRetired` remains for stories about the retired state itself.

packages/the-framework/src/e2e/harness.ts

@@ -1,4 +1,4 @@
-// The world behind the backend E2E story tests (see spec.md): the daemon's business logic wired
+// The world behind the backend E2E story tests (see README.md): the daemon's business logic wired
 // exactly as `runDaemon` wires it, against throwaway state, with runs spawned through
 // `fake-agent-bin.js` so the full production lifecycle executes offline.
 import { mkdtempSync } from 'node:fs'