Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions src/api/index.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3070,10 +3070,18 @@ describe("loops-api foundation", () => {
// later poll — that intent is correct and is covered by the "claim capacity"
// test above. But `dueSlots` under `catchUp: "latest"` returns ONLY the latest
// slot, so once wall time has moved past the wedged run's own slot the same-slot
// takeover it is being preserved for can never happen again: `overlap: "skip"`
// refuses the new slot because a `running` run exists, and the sweep skips that
// run because this runner owns it. Neither path can fire, so the loop is blocked
// for as long as the process lives.
// takeover it is being preserved for can never happen again, and the sweep skips
// that run because this runner owns it. Neither path can fire, so the run stays
// `running` behind a long-dead lease as an orphan row nothing can finalize, and
// the loop's cursor advances only if some later run happens to finalize — never
// through recovery.
//
// NOT "`overlap: "skip"` then blocks the loop": an EXPIRED lease does not, on its
// own, refuse the new slot. That gate turns on a run holding a LIVE lease or a
// live process (see the claim-sweep note in `src/api/index.ts`, and the store-level
// test "overlap skip does not block a later slot on an expired dead lease"). The
// defect under test here is the unreapable row and the recovery path, not a
// wedged scheduler.
//
// The existing "reclaims an expired overlap-skip lease" test does not reach this
// because it uses `catchUp: "all"`, which keeps the original slot in the due list
Expand Down Expand Up @@ -3134,7 +3142,9 @@ describe("loops-api foundation", () => {

// No phantom may survive: the run is either abandoned, or genuinely taken over
// with a lease in the future. What must not persist is `running` with a lease
// that expired in the past — that is the state which blocks `overlap: "skip"`.
// that expired in the past — the unreapable orphan row this defect leaves
// behind. (That row does not block `overlap: "skip"`; it is simply a row no
// path can finalize, so the loop's cursor never advances through recovery.)
const wedged = await storage.getRun(wedgedRunId);
expect(wedged).toBeTruthy();
const leaseStillExpired = wedged!.status === "running"
Expand Down
2 changes: 1 addition & 1 deletion src/cli/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2568,7 +2568,7 @@ hygiene
hygiene
.command("stuck")
.description(
"check or reclaim loop runs stuck 'running' with an expired lease and no live process (7cf8d8c1: overlap:skip then blocks the loop forever)",
"check or reclaim loop runs stuck 'running' with an expired lease and no live process (7cf8d8c1: an unreapable orphan row whose loop cursor never advances through recovery)",
)
.option("--apply", "abandon reclaimable runs and immediately advance their loop's nextRunAt")
.option("--limit <n>", "maximum runs to reclaim in one pass", "100")
Expand Down
15 changes: 13 additions & 2 deletions src/lib/hygiene.ts
Original file line number Diff line number Diff line change
Expand Up @@ -373,8 +373,19 @@ function toStuckRunEntry(run: LoopRun, reclaimed: boolean, deferredReason?: "liv
* Detect (and, with `apply`, reclaim) loop runs stuck in `status: "running"`
* with an expired lease and no live backing process — the `7cf8d8c1` defect
* class: a run that outlives both its lease and its execution timeout with no
* process behind it, which under `overlap: "skip"` blocks every later slot
* forever because nothing ever moves the run out of `running`.
* process behind it, leaving an unreapable orphan row whose loop's cursor
* never advances through recovery because nothing ever moves the run out of
* `running`.
*
* Be precise about what that state does NOT do, because the imprecise version
* ("`overlap: "skip"` then blocks the loop forever") sends the next reader to
* the scheduler instead of to recovery: an EXPIRED lease does not, on its own,
* refuse a later slot. That gate turns on a run holding a LIVE lease or a live
* process (`Store#hasBlockingRunningRunForOtherSlot`), which the store's own
* test "overlap skip does not block a later slot on an expired dead lease"
* (`src/lib/store.test.ts`) pins in exactly this shape. What this command
* repairs is the orphan row and the loop cursor behind it, not a wedged
* scheduler.
*
* The discriminator is evidence, not age: `Store#previewExpiredRunLeases` (and,
* on apply, `Store#recoverExpiredRunLeasesDetailed`) only ever classifies a run
Expand Down
Loading