Message signing runs synchronously on the Node.js main thread (blocks the event loop)

[thantsintoe]

Summary

Every outgoing message sent via send_with_header / multi_send_with_header is BLAKE2b-hashed and Ed25519-signed on the caller's thread — the Node.js main thread / event loop — inside the blocking NAPI call, before any work is handed to the tokio runtime. Signing is CPU-bound work that should run on a background thread, not on the event loop. Because it runs inline, every signature delays all other JS work (incoming message handlers, transaction processing, timers) on the single-threaded event loop.

Only the socket write (I/O) is currently offloaded to tokio. The CPU-heavy signing is not — which is backwards: the cheap part is offloaded, the expensive part is not.

Where it happens (code path)

1. NAPI entry point calls the sender directly in the function body, not inside RUNTIME.spawn:

   • shardus_net/src/lib.rs:391 — shardus_net_sender.multi_send_with_header(...) is called synchronously.
   • The RUNTIME.spawn above it (lib.rs:358) only wraps awaiting the send results and invoking the completion callback — not the signing.
   • The single-recipient send_with_header NAPI path has the same shape.

2. The sender serializes + signs inline before pushing to the async send channel:

   • shardus_net/src/shardus_net_sender.rs:83-98 (multi_send_with_header) and :69-80 (send_with_header) both call message.serialize_optimized(...) synchronously, then push the already-serialized bytes onto send_channel.
   • Only the socket write is spawned onto tokio (spawn_sender, sender.rs:122-152).

3. serialize_optimized is the CPU work:

   • shardus_net/src/message.rs:54-100 → crypto.hashslice(...) then crypto.sign(...).
   • crypto/src/lib.rs:95 hashslice → sodiumoxide::crypto::generichash = BLAKE2b.
   • crypto/src/lib.rs:118 sign → sodiumoxide::crypto::sign::sign = Ed25519 (SHA-512 internally).

(The plain send path at sender.rs:61-66 does not sign — only the header paths do. In practice shardus-core uses the header paths for essentially all traffic.)

Evidence

A V8 CPU profile of a node under load shows a large native crypto frame as a child of the JS multiSendWithHeader frame, consuming main-thread time. A V8 profile can only see the main thread — it is blind to tokio background threads. So the mere fact that this crypto work appears in the JS call tree is proof that it is executing on the event loop. If it were on a tokio worker, it would not appear in the JS profile at all.

Note on the symbol name: DevTools labels this frame crypto_stream_chacha20_ietf_xor_final. That exact symbol does not exist in the compiled .node binary (verified with nm/strings), and neither BLAKE2b nor Ed25519 use a ChaCha20 stream cipher. libsodium is statically linked with local-only symbols, so DevTools mis-symbolicates the native frame to a nearby name. The real work is the Ed25519 sign + BLAKE2b hash from serialize_optimized. A native profiler (macOS Instruments / dtrace / a perf build) can confirm the exact instruction breakdown, but the call path is unambiguous — serialize_optimized is the only crypto on this synchronous branch.

Impact

• Ed25519 sign + BLAKE2b hash is on the order of hundreds of microseconds per message batch. At low message rates (e.g. a 10-node local net at ~10 TPS) this is negligible and hides — the event loop has idle time to absorb it.
• Signing happens on every outgoing message batch, and message volume grows with both TPS and network size. At larger networks / higher TPS the accumulated per-message signing time keeps the event loop busy, delaying message handlers, the transaction-processing loop, and timers. This manifests as event-loop lag / send-resolution lag rather than as an obvious "crypto is slow" symptom.

Proposed fix

Move serialize_optimized (the hash + sign) off the calling thread onto a tokio worker, so it no longer runs on the Node.js event loop. Because signing is CPU-bound, spawn_blocking (tokio's blocking pool) is the appropriate primitive — not a regular async task.

Sketch (in ShardusNetSender::multi_send_with_header, sender.rs):

pub fn multi_send_with_header(&self, addresses: Vec<SocketAddr>, header_version: u8,
                              mut header: Header, data: Vec<u8>,
                              tx: mpsc::UnboundedSender<SendResult>) {
    let key_pair = self.key_pair.clone();
    let send_channel = self.send_channel.clone();

    RUNTIME.spawn_blocking(move || {
        header.set_message_length(data.len() as u32);
        let serialized_header = header_serialize_factory(header_version, header)
            .expect("Failed to serialize header");
        let message = Message::new_unsigned(header_version, serialized_header, data);
        let crypto = shardus_crypto::get_shardus_crypto_instance();

        // hash + sign now runs on a blocking worker, NOT the JS event loop
        let serialized_message = Arc::new(message.serialize_optimized(&crypto, &key_pair));

        for address in addresses {
            let tx = tx.clone();
            send_channel
                .send((address, serialized_message.clone(),
                       ChannelTransmitterType::MpscUnboundedSender(tx)))
                .expect("Failed to send data with header to channel");
        }
    });
}

Apply the same change to send_with_header. The NAPI entry point then returns to JS immediately; the messages are still signed exactly as before, just on a worker thread.

Notes / things to preserve

• Keep sign-once-per-batch. The current code signs the message once and shares Arc<serialized_message> across all recipients — do not regress this into per-recipient signing. The sketch above preserves it (sign once inside the closure, then fan out).
• Ordering. Moving the enqueue into a spawned task can reorder sends relative to each other. These routes are one-way tells/gossip that are already fire-and-forget and order-independent, so this should be safe — please confirm for any route that assumes ordering.
• key_pair is already a struct field; the crypto instance is a global getter — both are usable inside the closure.

Verification

• Re-capture a V8 CPU profile under the same load; the crypto frame under multiSendWithHeader should disappear from the JS call tree (it moves to a background thread the JS profiler cannot see).
• Event-loop responsiveness under load should improve (measure event-loop lag, or send-callback resolution latency, before/after).

Related (same package, separate issues)

• 5-minute uncleared timer leak in build/src/index.js _sendAug: for timeout === 0 sends, a 300s setTimeout is armed and never cleared on completion, pinning each message's payload for 5 minutes (heap snapshots show ~11k pending timers and hundreds of MB of retained payload buffers). Likely already tracked — cross-referencing since it lives in the same send path.
• JSON-stringify of already-binary payloads at the lib-net boundary (_sendAug runs jsonStringify(augData, ...), base64-encoding the binary payload) — a separate throughput issue on the same path.