feat: concurrency gate for mind turns — per-mind serialization by default, global cap

[psamiton]

Incident

On 2026-07-21 16:00:45, three schedules fired in the same second on bardo (mimsy heartbeat, whorl schedule, pip heartbeat — cron-aligned to the top of the hour). Three SDK sessions spun up and wrote concurrently; ten minutes later the kernel logged its first hung task — mmc_rescan blocked >120s on __mmc_claim_host (the SD card stalling under concurrent write load, no I/O errors). The host was intermittently starved for ~25 hours — TCP accepted on ports 22/1618 but neither sshd nor the daemon could respond — until a power cycle on 2026-07-22. A busy group conversation fanning out to all minds at once (fan-out: skipping hecate… × 33 in the final hour) produced the same pattern.

The trigger isn't specifically cron: any event that activates several minds at once (a #system message, a busy channel, a wake flush) starts N concurrent SDK subprocesses with no limit anywhere in the system. Slow storage turns that into a death spiral: writes back up → every turn slows down → turns overlap that wouldn't have → more concurrency.

Current behavior

• fanOutToMinds (packages/daemon/src/lib/delivery/fan-out.ts) delivers to every running mind participant immediately, in parallel, fire-and-forget.
• Within one mind, each session owns a resident SDK subprocess (templates/claude/src/agent.ts); turns are serialized within a session (streaming input channel) but sessions run concurrently, so a mind whose routes map channels to different sessions (or uses $new) can run several turns at once.
• Daemon-side, DeliveryManager already persists every delivery to delivery_queue before the POST, sweeps pending rows via redrive(), tracks per-(mind, session) active counts (incrementActive/decrementActive), and gets a synchronous sessionDone() on every turn end. turn-tracker.ts holds the activeTurns map. Nothing consults any of this as a limit.

Proposal: a concurrency gate in DeliveryManager, two scopes

One mechanism, both knobs — a gate check in the immediate-delivery path (deliverToMind / just before the POST):

1. Per-mind serialization (default on). If the target mind has an active turn in a different session, don't POST — leave the row pending. Deliveries to the session that is already mid-turn pass through (they fold into the current turn via the streaming input channel; they don't start a new subprocess). Net effect: a mind never runs more than one concurrent turn unless explicitly configured otherwise.
2. Global cap (configurable). If the number of active turns across all minds is at the cap, hold the row the same way. limits.globalConcurrentTurns in GlobalConfig (unset = unlimited); a Pi-on-SD-card host would set 2.

Release path: sessionDone() triggers an immediate redrive() (today only releaseGated does this; the periodic sweep would otherwise add up to REDRIVE_INTERVAL_MS of latency to every handoff). Redrive processes rows in id order, so held messages deliver FIFO as slots free.

Design notes

• Held ≠ failed. A gate hold must not increment the row's attempt count or trip dead-letter/backoff logic — it's a scheduling decision, not a delivery failure.
• Interrupts pass. A delivery with interrupt targets the active session's current turn by design; it folds in and doesn't add concurrency.
• Wake flush + batch flush go through the same choke point, so a sleeping mind's queued backlog and cron-aligned schedules serialize for free. Verify scheduler and system-event (deliverEvent immediate) paths also route through the gate or at least register in the active-turn accounting.
• Fairness/starvation: FIFO by queue id is fine at this scale; a chatty channel can't starve a DM because the DM's row is ahead of later chatter.
• Observability: log when the gate holds and for how long; surface current active-turn count in volute status.

Non-goals

• Staggering heartbeat crons — with the gate, aligned crons become harmless (they queue). The alignment isn't the root problem; unbounded concurrency is.
• Limiting resident idle SDK subprocesses (memory, not I/O) — the idle session reaper (perf(claude-template): reap idle sessions — shut down SDK subprocesses after idle timeout, resume on demand #458) already covers that dimension.

Evidence

Kernel log from the wedge (boot -3, bardo):

Jul 21 16:00:45 volute: fired "heartbeat" for mimsy
Jul 21 16:00:45 volute: fired "schedule-1771592422227" for whorl
Jul 21 16:00:45 volute: fired "heartbeat" for pip
Jul 21 16:10:24 kernel: INFO: task kworker/2:2:2723779 blocked for more than 120 seconds.
Jul 21 16:10:25 kernel: Workqueue: events_freezable mmc_rescan
Jul 21 16:10:25 kernel:  __mmc_claim_host+0xe4/0x218