core(supervision): rebuild from kameo reference — links, death-watch, restart strategies

[joeldsouzax]

Part of the core-rebuild epic — follows the shared rigor contract there.

Component

src/supervision.rs, src/links.rs (~2,588 LOC). Link graph, death-watch, SupervisionStrategy, restart. The gnarliest, most concurrency-heavy part — and the source of the current flaky-test pain (#100-class settle races).

Learning & discussion

Erlang/OTP-style supervision. Death detection and restart policy are distinct — Zenoh liveliness will later supply detection across nodes, but the policy state machine stays ours. Restart storms and link/unlink/die interleavings are where correctness lives.

Existing crates

parking_lot for the link graph locking (#52); consider petgraph for the graph structure.

Research — best algorithm/approach

OTP supervision trees; restart strategies (one-for-one / all-for-one / rest-for-one); escalation; restart-intensity/max-restarts limits (limit hit = Result, not panic).

Testing — hard as fuck

Sequence (link → die → notify ordering), lifecycle, linearizability (restart storms; concurrent link/unlink/die), and the heaviest DST burden of any card — deterministically reproduce the #100-class flakes instead of settling-with-timeouts. loom on the link graph. Mutation: cargo-mutants, zero survivors.

Absorbs

#52 (parking_lot for the supervision link graph).

[joeldsouzax]

Supervision surface + restart policy (from the #116 finalization research)

Two things land here from the run-loop research thread (see #116's finalized note).

The Supervisor trait — moved OFF base Actor

Per the actor-vs-supervisor split (OTP separate supervisor behaviour; actix separate Supervised trait; Armstrong thesis §4.3.2 "Workers and supervisors" + §5.2; POLA — watch/link/kill is more authority than send), supervision + linking are a separate supertrait, not base-Actor methods:

trait Supervisor: Actor {
    fn supervision_strategy() -> SupervisionStrategy { SupervisionStrategy::OneForOne }
    fn on_link_died(&mut self, actor_ref: WeakActorRef<Self>, id: ActorId, reason: ActorStopReason)
        -> impl Future<Output = Result<ControlFlow<ActorStopReason>, Self::Error>> + Send;
}

Dispatch on a spawn_linked / spawn_supervised entry point requiring A: Actor + Supervisor (the actix model) — keeps the base run-loop single, object-safe, and zero-cost for the common leaf actor that supervises nothing. ActorStopReason gains SupervisorRestart | LinkDied here (owned by #113).

Why separate, not default-methods-on-Actor: on_link_died's impl Future + generic Self makes the trait non-dyn-erasable — bundling it onto base Actor blocks dyn Actor for every actor. Quarantining keeps the base trait lean and erasable. (3 of 4 surveyed frameworks separate; only ractor/kameo bundle.)

Restart policy — two rules from Armstrong §8 (AXD301 field experience)

• Cascading restarts do not work. Restart the failed unit, do not cascade upward. (Wiger/AXD301: "restarting a failed process with the same arguments often worked, but cascading restarts generally did not help.")
• Bound restart frequency. After N restarts in T seconds the supervisor gives up and escalates (Armstrong's example: 5 restarts / 1000 s). Prevents restart storms.
• The strategy vocabulary (OneForOne/OneForAll/RestForOne) is a supervisor property, not an actor property — in OTP it lives in the supervisor's init, never in the child.

Recovery = rebuild, not resume

On child failure the supervisor restarts from a known-good state (Armstrong §8.3.1, Gray's conjecture: "most production faults are soft; reinitialize and retry"). For a bombay aggregate that's rehydrate from the nexus event log / snapshot (#12/#59), not resume the suspect in-memory state (Agha: a failed atomic step yields no valid successor behavior).

Cross-links: base lifecycle trait #116 · rebuild-from-log #12/#59 · identity-first ActorId #121.

[joeldsouzax]

⚠️ Implementation watch-outs (from the adversarial review #122)

• Erasure lives HERE, and that's correct (#10). The claim "no dyn anywhere" is false — a supervisor holding heterogeneous children must signal them without knowing A. Erasure is confined to the supervision graph (Box<dyn SignalMailbox> edges, built from weak senders so watching doesn't pin the child) + Recipient/group handles. This is off the message hot path (link/lifecycle rate only) — matches kameo & ractor. Don't try to eliminate it.
• Death-watch is PUSH, not poll (design: link-death — per-actor liveliness vs per-node + explicit death #8, post-arena-reversal). The dying actor's on_stop notifies its link edges; a watcher's WeakSender/is_closed() is the pull backup. There is no generational-slab staleness check (arena dropped, core(actor_ref): rebuild from kameo reference — ActorRef / Weak / Recipient #117).
• Restart rebuilds, never resumes. For an aggregate = rehydrate from the nexus event log/snapshot; for a non-aggregate = re-run on_start from Args. Never resume the suspect &mut self.
• No cascading restarts; bound restart frequency (Armstrong/AXD301: e.g. 5 restarts / 1000 s, then escalate). Strategy (OneForOne/…) is a supervisor property, not an actor property.

[joeldsouzax]

🧪 Test cases to write first (TDD — implement with this card)

• supervisor_signals_heterogeneous_children (sequence) — a supervisor with children of two DIFFERENT actor types sends Stop to all via the erased Box<dyn SignalMailbox> edges; assert all stop. Confirms the erased supervision path (the honest scope of erasure, epic: rebuild the runtime core part-by-part (kameo = reference oracle) #122-#10).
• @bug restart_rebuilds_never_resumes (lifecycle) — a child panics; on restart, its state is rebuilt (aggregate → replay log; non-aggregate → on_start from Args), never the torn &mut self. Fails if a supervisor resumes suspect state.
• restart_frequency_bounded_then_escalates (sequence) — N restarts within T → the supervisor gives up/escalates (Armstrong limit).
• no_cascading_restart (sequence) — a child failure restarts the child, not the parent subtree, under OneForOne.
• death_watch_push_notifies_links (sequence) — a dying actor's on_stop notifies watchers via weak signal-mailbox; a watcher receives LinkDied { id, reason }. (Ref epic: rebuild the runtime core part-by-part (kameo = reference oracle) #122-design: link-death — per-actor liveliness vs per-node + explicit death #8.)