Skip to content

Improvements identified by PilotSwarm's session-lifecycle work: shutdown quiescence, session lock timeout exposure, JS abort cancellation, and more #38

Description

@affandar

PilotSwarm recently implemented a session lifecycle protocol on top of duroxide (atomic per-turn state commits, warm session-affinity holds, versioned CAS snapshots) and built a literal fault-injection harness that kills real worker processes at protocol boundaries and lets duroxide's retry machinery recover them. That work surfaced seven concrete improvement opportunities in duroxide core, duroxide-pg, and the Node binding — all verified against duroxide 0.1.29 / duroxide-pg 0.1.34 sources.

Full write-up with code references, impact analysis, and fix shapes:
https://github.com/affandar/PilotSwarm/blob/feature/session-lifecycle-protocol/docs/bugreports/duroxide-improvements-20260705.md

Summary, ranked by value-per-effort:

  1. Runtime::shutdown should return on quiescence, not sleep the full budget (runtime/mod.rs ~1011). Dispatchers already quiesce early on the shutdown flag, but the call does an unconditional sleep(timeout_ms) before aborting leftovers — every graceful drain pays its full budget in wall-clock. Fix: tokio::time::timeout(budget, join_all(joins)) then abort, or a two-phase begin_drain() / await_quiescence().
  2. Expose the session lock timeout as a runtime option (duroxide-pg + node binding). The ~30s constant is the reclaim floor for session-pinned work after a worker crash — it dominates failover latency and makes kill/recovery test cycles ~30s slower each. workerLockTimeoutMs is already exposed; sessionLockTimeoutMs should mirror it.
  3. Aborting an activity does not cancel its JS execution (duroxide-node). JoinHandle::abort() orphans the JS promise; the activity body keeps running on the Node event loop after shutdown returns. Propagate cancellation (e.g. an AbortSignal in the activity context) or document + provide an await-settlement API.
  4. Quiescence observability: JsMetricsSnapshot has cumulative counters but no in-flight gauge, and the shutdown flag can't be set without committing to the fused blocking call.
  5. Document/test the retry-with-same-input contract: work-item retries re-deliver byte-identical input — application-level idempotency schemes hang off this; it deserves to be a stated guarantee. Nicety: expose the delivery attempt number in the activity context.
  6. Surface session lease lifecycle events (claimed/renewed/idle-expired/reclaimed) — affinity loss is currently silent to the host.
  7. (Design discussion) Optional session epochs — a lease-transfer epoch stamped on work items would give stateful-session applications the zombie-duplicate fence they currently must build themselves.

Happy to split these into individual issues and/or send PRs for 1–2, which are small and isolated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions