core(message): rebuild from kameo reference — Message trait + Context dispatch

[joeldsouzax]

Part of the core-rebuild epic — follows the shared rigor contract there.

Component

src/message.rs (~456 LOC). Message<T> (handle), Context (reply channel + actor ref), the CQRS message-vs-query split, type-erased DynMessage/BoxMessage.

Learning & discussion

Local tier: T: Send + 'static, no serialization. A message gets exclusive &mut self, processed sequentially; the query path can run reads concurrently. Context is how a handler replies / stops.

Existing crates

downcast-rs, dyn-clone (already used) — evaluate keeping vs a leaner type-erasure.

Research — best algorithm/approach

Type-erased dynamic dispatch patterns; CQRS command/query separation and its concurrency benefit; zero-cost handler wiring.

Testing — hard as fuck

Sequence/protocol (handler ordering, reply-exactly-once), lifecycle, defensive boundary (wrong-type downcast must not UB). DST over interleaved message/query. Mutation: cargo-mutants, zero survivors.

[joeldsouzax]

FINALIZED — message model: closed per-actor Msg enum, zero-alloc local mailbox

Researched across framework implementations (ractor / kameo / xtra / actix — exact code paths) + Rust idioms (enum layout, large_enum_variant, tokio oneshot). The model departs from all four frameworks, deliberately.

Honest headline (the surprise)

All four frameworks heap-allocate on every send — including ractor. ractor has the one-type Msg API, but its send path boxes the message to Box<dyn Any + Send> before the mpsc, because it goes through a type-erased ActorCell (heterogeneous registry). So zero-box is something no framework does — bombay pioneers it. The why is the design hinge: keep the send path monomorphized (typed ActorRef<A> → mpsc::Sender<Signal<A>>, Signal<A> a concrete enum, never a trait object). Ref-level erasure (Recipient/registry, #117/#119) holds a typed sender inside, so even an erased ref sends without boxing the message.

The model

• type Msg per actor = the aggregate's closed Command enum (event-sourcing already owns it). No open Handler<M>.
• Local mailbox = tokio::mpsc<Signal<A>> where Signal<A> is a concrete closed envelope enum: { Message(A::Msg), Stop, LinkDied{..}, /* system arms */ }. No Box<dyn> anywhere. Both layers closed → the loop is one match.
• tell = 0 heap allocations (move Signal::Message(msg) into the slot).
• ask = exactly 1 allocation — the oneshot Arc<Inner> — fully typed via a reply port (oneshot::Sender<Reply>) embedded in the requesting Msg variant (ractor RpcReplyPort pattern). Irreducible floor for a typed async reply; kameo pays this + boxed message + boxed reply (≥2 more).
• Large-variant trap: an mpsc<Signal> queues by value, so every slot costs size_of::<Signal>() = largest variant. Keep clippy large_enum_variant on; Box only the cold, fat variants (by measured frequency×size), never the whole enum. Zero-alloc stays for the common small messages.
• Reply typing: fully typed end-to-end (3 of 4 frameworks keep reply typed; only kameo erases the reply payload to Box<dyn Any>, forced by its single shared Signal<A>). We embed a typed oneshot::Sender<Reply> in the variant → no reply-side boxing.

Allocation, us vs the field

	local tell	local ask
bombay (this model)	0	1 (oneshot)
ractor / actix	1 box	1 box + 1 oneshot
kameo	1 box	1 box + 1 oneshot + 1 reply box
xtra	1 box + 1 oneshot (no tell fast-path)	same

Two-tier boundary

Local = Msg by value (zero-alloc hot path). Remote = serialized Command over Zenoh (nexus already defines the closed Command — reuse as the wire type), decoded and lifted into the same local Signal::Message(Msg) at the boundary via From/TryFrom. Allocation optimization is local-tier-only; on the wire it's serialization-bound, so a box there is noise. One hot loop, two entry points; the mailbox only ever sees Signal<A>.

What we give up (correctly)

External crates cannot add message variants to a closed enum (Rust enums are closed). kameo's open Handler<M> allows out-of-tree message injection — a feature you actively don't want at a single-writer consistency boundary. The command set is a domain contract.

Risk / validation

No framework ships the zero-box enum mailbox, so this is un-templated. The one real cost is enum-slot-size; mitigation is selective fat-variant boxing. Validate with a criterion bench (tell/ask alloc count + latency, enum vs boxed) before locking — the god-level bar + the bench card should carry it.

Knock-ons (the coordinated tightening)

• core(actor): rebuild from kameo reference — Actor trait, lifecycle hooks, run-loop #116 — trait gains type Msg; on_message(BoxMessage, Option<BoxReplySender>) -> Result<(), Box<dyn ReplyError>> collapses to handle(&mut self, Self::Msg, actor_ref, stop) -> Result<(), Self::Error>. All per-message Box/dyn leave the trait + loop.
• core(mailbox): rebuild from kameo reference — queue + Signal + backpressure #112 — mailbox = bounded tokio::mpsc<Signal<A>> over the concrete envelope enum; no Box<dyn>; this is the zero-alloc substrate.
• core(reply): rebuild from kameo reference — Reply + Delegated/Forwarded + ReplySender #115 — reply = typed oneshot::Sender<Reply> embedded per ask-variant; removes BoxReplySender / Box<dyn Any> reply erasure.
• core(actor_ref): rebuild from kameo reference — ActorRef / Weak / Recipient #117/core(registry): rebuild from kameo reference — local name to actor lookup #119 — Recipient/registry erasure holds a typed sender to preserve zero-message-box through an erased ref.

Sources

Rust Reference — enum layout · clippy large_enum_variant · enum_dispatch bench (enum ~4.5–10× vs Box<dyn>) · Ryhl — Actors with Tokio · ractor RpcReplyPort · tokio oneshot — single Arc<Inner> · kameo message.rs/mailbox.rs.

[joeldsouzax]

Revised by the adversarial review (#122)

Two cross-cutting seams the red-team surfaced (both must be decided here, as they thread through every signature):

• MaybeSend tier bound (#9): the actor/message API is Send-saturated and a single-threaded M6 client can't satisfy it. Abstract Send behind a cfg-gated MaybeSend alias (server = Send, client = empty) — or two traits — before core(mailbox): rebuild from kameo reference — queue + Signal + backpressure #112/core(actor_ref): rebuild from kameo reference — ActorRef / Weak / Recipient #117/core(registry): rebuild from kameo reference — local name to actor lookup #119 pick impls.
• Store trait seam (design: link-death — per-actor liveliness vs per-node + explicit death #8): the mailbox/ref/registry go behind traits so a deterministic impl carries loom/DST and a single-threaded impl carries the client tier (madsim/tokio precedent). The Msg-enum zero-alloc model itself is unchanged.

[joeldsouzax]

DECISION — #9 (MaybeSend): cfg-gated marker alias (option a) — LOCKED

The client tier hosts its own KERI identity actor (confirmed), so it runs a local single-threaded actor runtime — MaybeSend is required, not moot. The KERI identity actor is the canonical cross-tier case: the same actor must compile + run on a multi-threaded server (Send) AND a single-threaded constrained client (non-Send).

Decision: option (a) — a cfg-gated MaybeSend / MaybeSync marker alias.

• Threaded build (default / multi-thread feature): trait MaybeSend: Send {} + blanket impl<T: Send> MaybeSend for T {}. Here MaybeSend is Send — zero cost, byte-identical.
• Single-thread build (single-thread feature): trait MaybeSend {} + blanket impl<T> MaybeSend for T {} — the bound vanishes.
• Apply MaybeSend (+ MaybeSync where needed) in place of Send on: Actor, type Args, type Msg, type Error, M on sends, and every hook's impl Future + MaybeSend. The Arc↔Rc and tokio::mpsc↔local-queue choice is made by the same cfg via the Store/ref trait seam (epic: rebuild the runtime core part-by-part (kameo = reference oracle) #122-design: link-death — per-actor liveliness vs per-node + explicit death #8/#9).

Why (a), not (b)/(c): the Send/non-Send split is per-build-target, not per-platform — server + desktop are always Send; phone and IoT each straddle (native/Pi-class = Send; WASM, JS-bridge, or microcontroller = single-thread). One source must emit both builds; two-traits (b) / separate-surface (c) duplicate exactly where the straddlers live, and MaybeSend costs nothing on the Send builds.

Honest scope: MaybeSend is necessary but not sufficient for the microcontroller-IoT tier — that also needs no_std + a non-tokio executor (e.g. Embassy), reached via the same executor/Store trait seam. The WASM/JS-bridged phone needs single-thread but can keep std. So MaybeSend unlocks the actor trait for both tiers; the runtime under it swaps via the seam.

Canonical test: keri_identity_actor_compiles_and_runs_on_both_tiers — the same identity actor builds+runs under both the multi-thread and single-thread features (trybuild + a run test). This is the concrete form of client_tier_actor_compiles_single_threaded (#116), using the real cross-tier actor.

Cross-links: actor trait #116 · message model #114 · KERI identity actor #121 · trait seam/DST #122-#8 · review #122-#9.

[joeldsouzax]

Carry-over from #112 / the #122 design-risk note

The message-derive here must make the box-oversized-variants discipline first-class, not leave it to the large_enum_variant clippy lint (which only fires >200 B). Rationale + measured numbers in the #122 design-risk note: a fat inline Msg variant makes size_of::<Signal<A>>() = that variant, taxing every queue slot by value (measured 4104 B vs 16 B boxed → 256× for 1_000 queued msgs), and by-value send memcpys the whole message so zero-box inverts to pessimal above ~a cache line.

Concretely for #114:

• the derive should reject or auto-box Msg variants over a size_of budget, and emit a per-actor slot-size budget assert (like bombay-core's Signal size_of guard);
• per-message reply typing needs a typed oneshot::Sender<Reply> embedded per ask variant — keep those variants lean (they feed the slot-size tax).