Skip to content

docs(hygiene): correct the refuted overlap:skip mechanism in the stuck-run help and comments - #188

Merged
andrei-hasna merged 1 commit into
mainfrom
fix/0fb39457-hygiene-stuck-help-mechanism
Aug 3, 2026
Merged

docs(hygiene): correct the refuted overlap:skip mechanism in the stuck-run help and comments#188
andrei-hasna merged 1 commit into
mainfrom
fix/0fb39457-hygiene-stuck-help-mechanism

Conversation

@andrei-hasna

@andrei-hasna andrei-hasna commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

What

loops hygiene stuck --help asserts a mechanism that is factually wrong, and it shipped in @hasna/loops 0.4.38, so it is live in the help output on every machine that has installed it:

(7cf8d8c1: overlap:skip then blocks the loop forever)

An expired lease does not, on its own, block overlap: "skip". The correct mechanism was already in this codebase when the wrong string shipped — src/api/index.ts says it verbatim:

an expired lease does NOT block overlap: "skip". That gate refuses a new slot only while a run holds a LIVE lease or a live process (sqlite hasBlockingRunningRunForOtherSlot; the Postgres predicate is strictly more permissive still). So what this fixes is an unreapable orphan row and a recovery path that could not advance the loop — not a wedged scheduler.

The imprecision survived only in user-facing text and in test comments. This corrects it there.

Evidence — measured before changing anything

I did not take the premise on faith. Three-arm probe, in-memory sqlite only, so it can both pass and fail (raw output):

--- PROBE A: expired lease + dead process, live local daemon ticking ---
orphan slot          : 2026-01-01T00:10:00.000Z
tick at=2026-01-01T00:11:00Z claims=0 nextRunAt=2026-01-01T00:20:00.000Z orphanStatus=abandoned
tick at=2026-01-01T00:22:00Z claims=1 nextRunAt=2026-01-01T00:20:00.000Z orphanStatus=abandoned
tick at=2026-01-01T00:33:00Z claims=0 nextRunAt=2026-01-01T00:40:00.000Z orphanStatus=abandoned
hasRunningRun(final) : false
--- PROBE B: LIVE lease (control: overlap:skip gate with an unexpired lease) ---
orphan slot          : 2026-01-01T00:10:00.000Z
tick at=2026-01-01T00:11:00Z claims=0 nextRunAt=2026-01-01T00:10:00.000Z orphanStatus=running
tick at=2026-01-01T00:22:00Z claims=0 nextRunAt=2026-08-03T00:00:00.000Z orphanStatus=running
tick at=2026-01-01T00:33:00Z claims=0 nextRunAt=2026-08-03T00:00:00.000Z orphanStatus=running
hasRunningRun(final) : true
--- PROBE C: expired lease + dead process, NOTHING ticking ---
orphan slot   : 2026-01-01T00:10:00.000Z
nextRunAt     : 2026-01-01T00:10:00.000Z
orphanStatus  : running
hasRunningRun : true
  • A — the 7cf8d8c1 state under a ticking local daemon: the orphan is abandoned on the first tick, nextRunAt moves 00:10 -> 00:20 -> 00:40, and a fresh run is claimed at 00:22. Not blocked.
  • B — negative control, unexpired lease: claims=0 on every tick, hasRunningRun stays true. The gate does refuse while the lease is live, so A's result is attributable to the lease state and not to a broken probe.
  • C — nothing ticking: row stays running, nextRunAt frozen at the orphan slot. The unreapable orphan row and the stalled cursor, exactly as src/api/index.ts describes.

Third, independent confirmation already in the repo: the store's own behavioural test overlap skip does not block a later slot on an expired dead lease (src/lib/store.test.ts:266) pins the same fact.

Control provenance, stated explicitly: PROBE B validates the SUBJECT (the real overlap: "skip" gate on the code path hygiene stuck operates on), not merely the instrument. The grep sweep below carries its own positive control.

Changed — comments and one description string only, zero behaviour change

file what
src/cli/index.ts:2571 the shipped help string. 7cf8d8c1 reference preserved; still one line
src/lib/hygiene.ts:376 buildStuckRunReport doc comment, plus an explicit "NOT this" note quoting the refuted phrasing so it is recognisable rather than silently deleted
src/api/index.test.ts:3073 the site the #184 reviewer named as "the third artefact of record"
src/api/index.test.ts:3137 "that is the state which blocks overlap: skip"

Deliberately NOT changed — each is accurate, and PROBE B is why

These all describe a run whose process looks alive, where the gate genuinely does refuse:

  • src/lib/hygiene.ts"which under overlap: "skip" is exactly the 'blocks every run queued behind it' failure" (about preserveLiveProcesses on a live-looking run)
  • src/lib/store.ts:5071"deferring it forever blocks every run queued behind it" (same, live-looking)
  • src/lib/hygiene.ts"overlap: "skip" only re-admits new claims once no running run remains" — true of the scheduler-level gate Store#hasRunningRun, which is status-only
  • src/lib/hygiene.test.ts:259 and :269 — both keyed on hasRunningRun, same reason

Discriminator applied throughout: "a running run blocks overlap: skip" is true of the status-only hasRunningRun gate; "an expired-lease run blocks it" is false. Only the second was changed.

Gates

Captured by redirect to a file, exit status read from the command, nothing piped.

bun run typecheck   -> rc=0
bun run test        -> rc=0    1127 pass, 56 skip, 0 fail, 1183 tests / 74 files
bun run build       -> rc=0

A note on my own instrument, because it produced a false finding first. My initial run used bare bun test and reported 4 failures. All four were timed out after 5000ms — the package's own script is bun test --timeout 120000, so the bare form was the wrong instrument, and the number it printed was the budget, not a duration. I confirmed it against the untouched base sha d4628e67 before drawing any conclusion: the same tests time out identically there (5010.67ms vs 5011.71ms on the isolated case), i.e. pre-existing and load-sensitive, station load 14.51 on 20 cores. Under the real gate the suite is green.

No test asserts on the description string (grep for it across *.test.ts returns nothing; positive control: the same pattern matches in src/cli/index.ts, and grep does return hits in that test scope for other terms). So nothing needed adjusting to match the new text.

Verified in the built artifact, not just the source — bun dist/cli/index.js hygiene stuck --help prints:

check or reclaim loop runs stuck 'running' with an expired lease and no live
process (7cf8d8c1: an unreapable orphan row whose loop cursor never advances
through recovery)

Not published, and the wrong string stays live until it is

No version bump and no publish here. 0.4.38 shipped under an hour before this branch was cut. The correction reaches users only on the next release; until then loops hygiene stuck --help keeps printing the refuted mechanism on every installed machine.

What I did not check

  • The Postgres path was not exercised. src/api/index.ts calls its predicate "strictly more permissive still" than the sqlite one; I did not measure that, and I neither relied on it nor restated it as my own finding.
  • No hosted control-plane state was read or written, and loops hygiene stuck --apply was never run anywhere. Every probe used new Store(":memory:"). The only CLI invocation was --help, which touches no store.
  • docs/ and README.md were not modified — the grep sweep found no instance of the refuted claim in either.

Task: 0fb39457.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…k-run help and comments

`loops hygiene stuck --help` shipped in 0.4.38 asserting that a run stuck
'running' with an expired lease and no live process means "overlap:skip then
blocks the loop forever". That mechanism is wrong, and the correct one was
already in this codebase, in src/api/index.ts: an expired lease does not, on
its own, refuse a later slot — that gate turns on a run holding a LIVE lease
or a live process. What the defect actually leaves is an unreapable orphan
row whose loop cursor never advances through recovery.

Measured before changing anything, in-memory sqlite, three arms so the probe
can both pass and fail:

  A  expired lease + dead process, local daemon ticking
     -> orphan abandoned on the FIRST tick, nextRunAt 00:10 -> 00:20 -> 00:40,
        a fresh run claimed at 00:22. Not blocked.
  B  LIVE unexpired lease (negative control)
     -> claims=0 on every tick, hasRunningRun stays true. The gate does refuse
        while the lease is live, so arm A's result is the lease state and not
        a broken probe.
  C  expired lease + dead process, nothing ticking
     -> row stays 'running', nextRunAt frozen at the orphan slot. The orphan
        row and the stalled cursor, exactly.

The store's own test "overlap skip does not block a later slot on an expired
dead lease" (src/lib/store.test.ts) pins the same fact independently.

Changed (comments and one description string only, no behaviour):
  src/cli/index.ts        the shipped help string; 7cf8d8c1 reference kept
  src/lib/hygiene.ts      buildStuckRunReport doc comment, plus an explicit
                          "NOT this" note naming the refuted phrasing
  src/api/index.test.ts   two comments, incl. the site the #184 reviewer named

Deliberately NOT changed, because each describes a run whose process LOOKS
ALIVE, where the gate genuinely does refuse (arm B): hygiene.ts "blocks every
run queued behind it", store.ts:5071, and the hasRunningRun-keyed comments in
hygiene.test.ts.

No version bump and no publish: the wrong string stays live in 0.4.38 until
the next release.

Agent: agent-chief-planning
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

[REVIEW] GO — #188 @ 11b07bf — lens: docs-accuracy, reviewer Lucretius (1 of 1)

Independent read. I was given the questions, not the author's conclusion, and I measured
before reading the PR body. Every claim below is either raw output pasted verbatim or
labelled UNTESTED.

Q1 — what the defect state actually causes

It does NOT prevent an overlap: "skip" loop from claiming a later slot. Measured on
new Store(":memory:") only; nothing was run against a real loops store, and
hygiene stuck --apply was never invoked.

A EXPIRED-dead  first.status= running pid= null leaseExpiresAt= 2026-01-01T00:00:00.010Z | later-slot claim = CLAIMED
A' LIVE lease   first.status= running leaseExpiresAt= 2026-01-01T01:00:00.000Z | later-slot claim = REFUSED
B same dead row: hasRunningRun = true | hasBlockingRunningRunForOtherSlot = false
C claimDueRuns: claims= 1 skipped= 0 recovered= 1 | wedged row now = abandoned
C' claimDueRuns LIVE: claims= 0 skipped= 1 recovered= 0 | row now = running
D tick: nextRunAt 2026-01-01T00:01:00.000Z -> 2026-01-01T00:06:00.000Z

A/A' and C/C' are two-sided: the same probe both claims and refuses depending only on lease
state, so CLAIMED is attributable to the expired lease and not to a broken probe.

Surface by surface:

  • sqlite Store#claimRun — gates on hasBlockingRunningRunForOtherSlot
    (src/lib/store.ts:4311-4323), which returns true only for a row with `lease_expires_at

    now`, a live recorded pid, or a live workflow step process. The dead row passes straight
    through. Row A.

  • Postgres claimRunsrc/lib/storage/postgres-loop-storage.ts:1218-1227 requires
    lease_expires_at IS NOT NULL AND lease_expires_at > $3. Same answer, strictly more
    permissive (no pid check at all). UNTESTED live — read from source; exercising it needs
    a Postgres instance I did not stand up.
  • Local scheduler — this is the one place where a naive reading could go the other way,
    and it is worth stating explicitly: runSlot and claimSlot (src/lib/scheduler.ts:407
    and :463) gate on the status-only Store#hasRunningRun, which does count the dead
    row — row B shows both predicates disagreeing on the same row in the same instant.
    It still does not block, for a different reason: recoverAndExpire runs first in the same
    tick with no excludeClaimedBy/preserveLiveProcesses, so the row is abandoned before the
    gate is reached (C: recovered=1, row abandoned, claims=1). Control C' shows the
    gate genuinely refusing when the lease is live.

What the state does cause is the orphan row itself: running behind a dead lease that the
normal sweep skipped, and a loop cursor that never advances through recovery.

Q2 — is the new wording accurate, and the removed wording inaccurate?

Both directions check out.

  • Removed (overlap:skip then blocks the loop forever, and that is the state which blocks overlap: "skip") — refuted by A vs A'.
  • Added — accurate, and notably it names its gate (Store#hasBlockingRunningRunForOtherSlot)
    rather than saying "the overlap gate", which is what keeps it true.
  • The cited corroborating test exists and pins exactly this shape:
    src/lib/store.test.ts:266 test("overlap skip does not block a later slot on an expired dead lease")
    leaseMs: 10, no pid recorded, later-slot claim asserted running.
  • src/api/index.ts:1363-1369 already carried the correct mechanism at base d4628e67, so
    this commit makes the remaining artefacts consistent with prose that was already in-tree
    rather than inventing a new claim.

Q3 — is the "deliberately NOT changed" discrimination correct?

Yes. I swept for the refuted claim independently before reading the PR body and surfaced one
candidate the diff leaves alone:

src/lib/hygiene.ts:430"overlap: "skip" only re-admits new claims once no running run
remains for the loop"
.

Read against claimRun that sentence is the refuted claim in different clothes. Read against
hasRunningRun it is exactly true — which is precisely the split row B measures, and the
PR body already states that rationale verbatim. The discrimination is correct; I could not
find an instance carrying the refuted claim that was left behind, nor one changed that was
fine. Positive control for the sweep: the same grep shape returns the removed phrasing when
run against git show d4628e67:src/lib/hygiene.ts (376: * process behind it, which under overlap: "skip" blocks every later slot), so the probe can find the thing when it is there.

Q4 — does the diff change behaviour?

No program logic changes. Filter over every changed line excluding comments and the one
description string:

non-comment changed lines: 0
=== POSITIVE CONTROL: same filter over a known code-changing commit (6ccb634) ===
20

git diff --numstat: src/api/index.test.ts 15/5, src/cli/index.ts 1/1,
src/lib/hygiene.ts 13/2.

The description string is user-visible help text — that is the point of the change, not an
unstated side effect. No test asserts on it (grep for both the old and the new string across
*.test.ts returns nothing; positive controls: the old string matches 1 in
git show d4628e67:src/cli/index.ts, and grep over that same *.test.ts scope does return
hits for other hygiene-stuck terms).

Gates I ran myself, captured by redirect, exit status read from the command, nothing piped:

bun run typecheck                                              -> rc=0
bun test --timeout 120000 hygiene.test store.test scheduler.test -> rc=0   130 pass, 0 fail
bun test --timeout 120000 src/api/index.test.ts                  -> rc=0    65 pass, 0 fail

I ran the four files covering the changed surfaces, not the full 74-file suite — the PR
reports 1127 pass / 0 fail on the full run and I did not reproduce that number.

Non-blocking follow-ups (P3 — do not hold the merge)

  1. src/lib/hygiene.ts:430 is correct but unqualified in-file: the new text ~45 lines above
    names its gate, this one does not, so a reader landing there gets the status-only rule
    without being told it is the status-only rule. One clause (— the status-only Store#hasRunningRun gate) would close it. The PR body already reasons this way; the file
    does not.
  2. "unreapable orphan row" appears in the doc comment of the command that reaps it. Accurate
    in context (not reaped by the normal recovery sweep), mildly self-contradictory read cold.

Neither is a defect in what the commit asserts. Verdict stands: GO.

@andrei-hasna
andrei-hasna merged commit ac1cd76 into main Aug 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant