epic: rebuild the runtime core part-by-part (kameo = reference oracle)

[joeldsouzax]

Decision (#63)

kameo is a reference oracle, not a fork base. We rebuild the local runtime core in-place as bombay, part-by-part, each part guarded by the existing test suite (#74) and consulting kameo for behaviour. Identity is core, not behind a transport flag — the identity-first inversion (kameo is process-first/identity-incidental; we are identity-first/process-embodies-identity). Scaffolding (god-level clippy bar, CI, cucumber harness, macros, console) is kept. Timeline: as long as it takes — we hard-test every part.

Two-tier message model stays (see #66 design note): cheap local (non-serialized, in-memory) vs. serialized dataspace citizens. Only the backend of the remote tier changes (libp2p → Zenoh).

Per-card rigor contract (every sub-card follows this)

1. Learning & discussion — mental model; how kameo does it; what we keep vs. change.
2. Existing crates — survey + pick (folds in the de-handroll cards cleanup: adopt thiserror for local error enums #50–cleanup: tokio-util CancellationToken for task abort #55, cleanup: lock-free local registry (papaya / dashmap) #51).
3. Research — best algorithm/approach — primary sources (papers, prior art, arxiv) before committing to an impl. No commitment before the survey.
4. Deterministic Simulation Testing (DST) — seedable, replayable, interleaving-exploring.
5. Mutation testing — cargo-mutants; zero surviving mutants in the module.
6. Cross-cutting tests (CLAUDE.md rule 7: sequence/protocol · lifecycle · defensive boundary · linearizability) + property tests + loom where atomics/ordering live.
7. Definition of Done — rebuilt in-place in bombay (no chore: god-level clippy — adopt nexus elite lint config #61 quarantine header), god-level bar clean; existing + new tests green; DST seeds pass; zero surviving mutants; behaviour parity proven by the test net; identity-first where relevant.

Sub-cards (dependency order — mailbox first)

• core(mailbox): rebuild from kameo reference — queue + Signal + backpressure #112 mailbox — queue + Signal + backpressure (related feat: mailbox strategies via channel handlers (RingChannel keep-latest vs FifoChannel) #19, cleanup: tokio-util CancellationToken for task abort #55)
• core(error): rebuild from kameo reference — typed error domains #113 error model — typed error domains (absorbs cleanup: adopt thiserror for local error enums #50, cleanup: terrors OneOf for nested error unions #53)
• core(message): rebuild from kameo reference — Message trait + Context dispatch #114 message + Context — Message trait, dispatch
• core(reply): rebuild from kameo reference — Reply + Delegated/Forwarded + ReplySender #115 reply — Reply, Delegated/Forwarded, ReplySender
• core(actor): rebuild from kameo reference — Actor trait, lifecycle hooks, run-loop #116 actor + loop — trait, lifecycle hooks, run-loop (absorbs cleanup: tokio-util CancellationToken for task abort #55)
• core(actor_ref): rebuild from kameo reference — ActorRef / Weak / Recipient #117 actor_ref — ActorRef / Weak / Recipient
• core(request): rebuild from kameo reference — ask/tell builders #118 request — ask/tell builders (absorbs cleanup: bon for request builders #54)
• core(registry): rebuild from kameo reference — local name to actor lookup #119 local registry — name to actor lookup (absorbs cleanup: lock-free local registry (papaya / dashmap) #51)
• core(supervision): rebuild from kameo reference — links, death-watch, restart strategies #120 links + supervision — link graph, death-watch, restart (absorbs cleanup: parking_lot for the supervision link graph #52)
• core(identity): identity-first ActorId — AID/SAID/delegation, key-expr addressable #121 identity-first ActorId — AID/SAID/delegation, key-expr (reframes refactor: ActorId identity — libp2p PeerId to Zenoh key-expr #2)

No duplicates: de-handroll cards (#2, #50-#55, #51) are folded into the matching sub-card as acceptance criteria and closed.

[joeldsouzax]

Adversarial review of the core spine — 13 problems verified, resolved, and the ref-model reversal

Red-teamed the finalized core (#112–#121) against the chosen stack. Five parallel streams, each verifying from primary source (Rust nomicon/std/tokio/sharded-slab/kameo source) + researching the fix within our crates. All 13 problems are real; the fixes are below. The headline: #13 reverses #117 to a kameo-style Arc/Weak hybrid, which dissolves #1/#5/#6/#7 and improves #8/#9.

The reversal (#13) — drop the generational arena

The arena's pitched unique benefit — "a cheap Copy, liveness-bearing key that doesn't pin the channel" — is false: std Weak and tokio WeakSender already give non-pinning liveness (upgrade()/is_closed()), and kameo already builds its watch/link edges on downgraded weak senders (weak_signal_mailbox → Box<dyn SignalMailbox>). The arena's real marginal benefit reduces to Copy (no atomic RMW) + cache-locality + dense integer index — which pays off only for a large, hot, frequently re-cloned watcher set (>~1e4 tokens), not the event-sourced-aggregate common case. Against that unproven benefit it drags in: the panic-safe Drop-guard reap (#1), guard-across-await spin (#5), 13-bit ABA in ~13.6 min on churny slots (#6), MAX_THREADS panic (#7), loom/DST opacity, and no no_std → cannot serve the M6 pico tier at all. Also semantically it's poll-based liveness (generation diff) vs kameo's push model (dying actor's on_stop notifies links) — a different architecture, not a free swap.

Decision: HYBRID. ActorRef<A> hot path + own liveness = kameo Arc<Sender>/WeakSender (typed mpsc::Sender<Signal<A>>, is_alive() via is_closed(), WeakActorRef = WeakSender for non-pinning watch). Registry (#119) keeps a concurrent map (papaya) of AID → erased weak handle, behind the trait seam. No sharded-slab — gate it behind benchmarks (local-resolve throughput; watcher fan-out clone+drop at N=10..1e4; churn correctness) and add only if a specific bottleneck is measured (rule #4: second concrete use).

The 13-point verdict

#	Verified	Resolution (within our stack)
1 panic leaks slot	real	dissolved (no slot in hybrid). General principle kept: death-critical teardown = RAII Drop, panic-free (double-panic→abort), never user code.
2 reply can't carry HandlerError	real — spec mixed ractor+actix models	reply port = oneshot::Sender<Result<Reply, E>>; handler sends exactly one Ok(reply)/Err(e); handler's -> Result<(), Self::Error> return is reserved for supervision (poison/notify), NOT the asker's domain error. Ask-side map: Ok(Ok)→Ok, Ok(Err(e))→HandlerError(e), Err(RecvError)→InterruptedBeforeReply.
3 died-before-reply	real — no variant	add InterruptedBeforeReply (= ractor CallResult::SenderError), from the oneshot RecvError; carries no M (already consumed); own retryability class.
4 bounded-mailbox cycle deadlock	real (all 4 Coffman conditions)	cannot be made structurally impossible while a handler may send().await. Only structural prevention = CPS: never block the message loop (outbound asks fire-and-forget, reply returns as a new self-message). Otherwise try_send+MailboxFull / ask-timeout = detection, not prevention. Decision owed on #118.
5 guard spins worker	real (take spins on std::thread::yield_now)	dissolved (no slab). General: if a slab is ever added, guard-hiding wrapper + remove not take.
6 13-bit gen ABA	real & reachable (period 8191, ~13.6 min at 10 cyc/s; watcher path unprotected)	dissolved (no generation). If ever added: 32-gen-bit Config + per-spawn ULID in slot metadata.
7 shard insert-full / MAX_THREADS panic	real (panic is the sharp edge)	dissolved.
8 DST/loom opacity	real — papaya loom-opaque; sharded-slab loom-capable; seize is hyaline (no bg thread)	Store/Registry trait seam; model-checked logic in a deterministic impl (slotmap+HashMap+loom atoms); prod impls ride upstream loom + integration tests. Precedent: madsim/tokio. Hybrid makes the ref path (Arc/Weak/mpsc) loom-visible.
9 std/threads block M6 client	real	same trait seam per tier (client = Rc+slotmap+local queue on LocalSet). NEW blocker: the landed Actor trait is Send-saturated (Actor: Send, Args: Send, hooks +Send, M: Send, Arc in ActorRef) → a single-threaded client can't satisfy it. Needs a tier-abstracted MaybeSend bound (or two traits) decided on #114/#116 BEFORE impls.
10 "no dyn anywhere" false	real	adopt scoped truth: no dyn on the message hot path; erasure is confined to the supervision graph (Box<dyn SignalMailbox>) + Recipient/group (Box<dyn MessageHandler<M>>), touched at link/lifecycle rate. Matches kameo & ractor.
11 monomorphization	real but noise	tens of aggregate types ≈ 1–2 MB .text; keep the loop monomorphized; keep kameo's erased spawn/restart seam; measure with cargo bloat/llvm-lines only if N→hundreds.
12 on_stop on torn state	real (logic bug, not UB)	run on_stop after Panicked (OTP terminate precedent), pass the reason, treat &mut self as poisoned (std Mutex-poison analogy) — reason-independent resource release only, never flush/derive from domain fields; critical teardown stays in RAII.

What changes on which card

• core(actor_ref): rebuild from kameo reference — ActorRef / Weak / Recipient #117 ref — REVERSED to kameo Arc/WeakSender hybrid; sharded-slab dropped (benchmark-gated). Supersedes the sharded-slab finalization.
• design: link-death — per-actor liveliness vs per-node + explicit death #8 death-watch — REVERSED: push (dying actor notifies links via weak signal-mailbox) + pull is_closed() backup; not poll-staleness. Remote = Zenoh liveliness token (unchanged).
• core(identity): identity-first ActorId — AID/SAID/delegation, key-expr addressable #121 identity — local id = a generated id (kameo-style) / the AID, not a generational slab key; liveness is channel-closed + push, not generation. Two-tier (local id ↔ global AID) + migration=rotation unchanged.
• core(actor): rebuild from kameo reference — Actor trait, lifecycle hooks, run-loop #116 loop — teardown simplifies (no slot reap); ADD poisoned on_stop after panic (feat: queryable + persistent actor state via Zenoh storages #12); the RAII-for-critical-teardown principle (kept lightweight); the MaybeSend tier bound (#9); the Store trait seam (design: link-death — per-actor liveliness vs per-node + explicit death #8).
• core(error): rebuild from kameo reference — typed error domains #113 error — ADD InterruptedBeforeReply variant (feat: replace DHT registry with key-expr discovery #3); note the two error channels (refactor: ActorId identity — libp2p PeerId to Zenoh key-expr #2).
• core(reply): rebuild from kameo reference — Reply + Delegated/Forwarded + ReplySender #115 reply — reply port = oneshot::Sender<Result<Reply, E>> (refactor: ActorId identity — libp2p PeerId to Zenoh key-expr #2); handler sends Ok/Err; handler return = supervision.
• core(request): rebuild from kameo reference — ask/tell builders #118 request — DECISION owed: CPS vs blocking-ask+timeout (feat: messaging over Zenoh query/reply #4); the ask-side reply mapping (refactor: ActorId identity — libp2p PeerId to Zenoh key-expr #2/feat: replace DHT registry with key-expr discovery #3).
• core(message): rebuild from kameo reference — Message trait + Context dispatch #114 message / core(mailbox): rebuild from kameo reference — queue + Signal + backpressure #112 mailbox — the MaybeSend bound (#9) + Store/Registry trait seam (design: link-death — per-actor liveliness vs per-node + explicit death #8/#9).
• core(registry): rebuild from kameo reference — local name to actor lookup #119 registry — papaya kept, stores erased weak handles; behind the trait seam.

Net

The critical pass removed a subsystem (the arena) rather than adding complexity — the strongest possible outcome. Remaining genuine work is small and local: the reply-port Result wiring, one new error variant, the deadlock posture decision, the MaybeSend/trait-seam abstraction for DST + the client tier, and the poisoned-on_stop contract. None require a crate not already chosen.