Finishing the Binary Migration

[thantsintoe]

1. What lib-net is and what it actually achieves today

lib-net is the network transport for all internal node-to-node traffic. It is a Rust library (neon/NAPI bindings, shardus_net/src/) plus a TypeScript wrapper (build/src/index.js).

The Rust side owns:

• TCP sockets: connection setup, pooling, and reuse (shardus_net_sender.rs, shardus_net_listener.rs)
• Wire framing and a versioned header mechanism (header_factory.rs, header/)
• Optional per-message compression — gzip and brotli are implemented and selectable via header flag (compression.rs)
• Message signing/verification support (shardus_crypto.rs) and transfer stats (stats.rs)
• A tokio runtime so all socket I/O runs on Rust threads, off the JS event loop

The TS wrapper owns:

• The augData envelope (UUID, ports, timestamps, msgDir — types.js:NewAugData)
• Request/response correlation (responseUUIDMapping) and timeout timers
• Serialization of the outgoing message to a string, using a stringifier injected by shardus-core (Utils.safeStringify)

What it demonstrably achieves: in the local 10-node test, the JS main thread was only 6–7% busy while the node's process CPU ran several times higher (~29%) — the socket and I/O work genuinely lives on Rust threads. (Both figures come from a shared laptop running all 10 nodes; the ratio is the evidence here, not the absolute percentages.) Connection reuse, framing, and request correlation all work; the network delivered ~10 TPS with 1.0 vote sends per tx and zero timeouts. lib-net's core job — keep networking off the JS thread — is being done.

What it costs today: two defects, one known and one structural:

1. (Known — fix in flight) For one-way sends (timeout === 0, i.e. every tell and gossip), _sendAug arms a 300-second setTimeout that is never cleared when the send-complete callback resolves (build/src/index.js:146-150). Each timer pins its closure context — including the serialized payload — for 5 minutes. Measured: 11,328 pending timers and ~94MB of pinned payload buffers on a local node; extrapolates to the ~490MB arrayBuffers on cloud nodes.
2. (Structural — the subject of this plan) The JSON-in-the-middle problem, below.

---

2. The JSON-in-the-middle problem

2.1 The contract that froze

Every native send function accepts the message body as a string, and the listener delivers inbound messages to JS as a string:

• send: cx.argument::<JsString>(2) — shardus_net/src/lib.rs:185
• send_with_header: cx.argument::<JsString>(4) — lib.rs:236
• multi_send_with_header: cx.argument::<JsString>(4) — lib.rs:314
• Listener channel: UnboundedReceiver<(String, SocketAddr)> — lib.rs:118

There is no raw-bytes lane. This contract was designed when every payload was a JSON object anyway, so "string in, string out" was free. Then shardus-core's binary serialization project happened — compact typed buffers (VectorBufferStream), binary/* routes, typed readers — but the NAPI contract was never upgraded. The binary migration stopped at the lib-net boundary.

2.2 What one "binary" tell actually goes through

#	Step	Where	Cost
1	Typed message → compact binary Buffer	shardus-core serializer (Comms.tellBinary)	intended cost (cheap, schema-driven)
2	Buffer wrapped as {route, payload} → augData envelope	network/index.ts:326, lib-net NewAugData	object alloc
3	safeStringify(augData): Buffer → base64 string ({"dataType":"u8ab","value":"…"}) + JSON envelope around it	lib-net _sendAug (index.js:81), encoding in @shardus/lib-types stringify.js:118-129	base64 encode (touches every payload byte) + stringify walk + one large string alloc
4	JsString → Rust String	NAPI boundary	full UTF-8 copy of the whole message
5	Framed, optionally compressed, sent	Rust threads	compression works on base64 text, not bytes (worse ratio, more CPU)
6	Receive: bytes → Rust String → JS string	listener, NAPI	another full copy + JS string alloc
7	jsonParse with type reviver: JSON parse + base64 decode → new Buffer	lib-net extractUUIDHandleData	parse + decode (touches every byte again) + Buffer alloc
8	Typed reader consumes the Buffer	shardus-core deserializer	intended cost

Steps 1 and 8 are the binary serialization project. Steps 3–7 are the old JSON world wrapped around it. The payload bytes are touched/copied roughly 6–8 times per hop in string or encoded form; the design intent was ~2.

On the non-combined send path (useCombinedTellBinary: false), step 3 repeats per recipient for the identical payload — a gossip batch to 3 peers stringifies the same message 3 times.

2.3 How this shows up as CPU

• JS main thread (local profile — directional, not calibrated): safeJsonParse/stringify frames ≈ 16% of the JS thread's busy time, with extractUUIDHandleData and node:buffer frames also present. Treat 16% as a floor, for two reasons. First, the profile comes from a MacBook running all 10 nodes plus unrelated software, so scheduler contention distorts every absolute figure. Second, it was captured under the mildest message conditions this code path ever sees — 10-node group, fanout-3 gossip, small app payloads, ~11 message batches per tx. Per-tx message count and payload bytes both grow with group size (corresponding tells, receipt signature packs, wider gossip), so the JSON/base64 share of JS work at 32 nodes should be materially higher. No CPU profile of a 32-node node exists yet; capturing one on representative hardware is a prerequisite for calibrated numbers (§4).
• Native/other threads (inferred; both test environments were contended): the reliable signal here is the shape, not the magnitudes. The JS thread is nearly idle (~6% busy) while the node process's CPU is several times higher (~29% locally), and the frames that do appear on the JS thread are exactly the §2.2 steps — which places the remainder (string copies across NAPI, framing, compression of inflated text, V8 background GC) on threads JS-level profiling cannot see. That invisibility is precisely why this cost went unnoticed. The absolute percentages — including the cloud run's ~56% at 12 TPS and the "saturates around 20–22 TPS" extrapolation in the TPS plan — come from hosts running multiple node processes, so they are trend lines for that specific hardware, not production calibrations.
• GC pressure: every message allocates an intermediate base64 string, a full-message JSON string, and (on receive) parsed objects plus a fresh Buffer. These short-lived megabyte-scale allocations are exactly the population the uncleared timers were pinning; even after that fix, they remain per-message garbage. GC was ~12% of JS busy locally.
• Wire size: base64 inflates payload bytes by ~33%, plus the JSON envelope ({"dataType":"u8ab","value":…} wrapper, augData fields as JSON). More bytes = more copy work in every one of steps 4–7, and worse compression.
• Scale coupling: message batches per tx per node ≈ 11–12 (local, 10 nodes) and the count grows with group size; cloud runs ~200 outbound batches/s/node at 12 TPS. The per-message tax multiplies with exactly the variables we want to grow (TPS × group size × payload size).

2.4 What survives despite the problem (why binary wasn't pointless)

• Wire size is still ~30–50% better than the pre-binary nested-JSON format for hash-heavy messages (32-byte hash = 44 base64 chars vs 64 hex chars + field names).
• Receive-side JSON.parse of an envelope whose bulk is one long string token is far cheaper than parsing deep object JSON into thousands of heap objects, and AJV validation is gone.
• The typed readers and versioned routes are a correctness win independent of perf.

The binary project delivered roughly half its intended win. This plan is about collecting the other half.

---

3. Proposed changes

Ordered by impact-per-risk. Items 1–2 are the heart of the plan.

3.1 Fix send-promise lifecycle (P0 — leak + backlog; partially in flight)

1. Keep the timer reference in the timeout === 0 branch of _sendAug and clearTimeout it inside sendCallbackMk3/multiSendCallback; cap the fallback at ≤10s instead of 300s.
2. Investigate the ~4,900 suspended _wrappedSendAug frames (sends whose native callback never fired or fired minutes late): audit Rust callback dispatch on every error path, and the awaitProcessing semantics of multi_send_with_header (one slow peer must not hold the whole batch's callback).
3. In shardus-core, stop awaiting Promise.all over per-peer sends for fire-and-forget routes; report per-peer failures via counters instead.

CPU effect: indirect but real — removes hundreds of MB of pinned heap (GC scans shrink), and stops the event-loop from carrying tens of thousands of live timers and suspended frames. Verifiable immediately: pending Timeout count should drop from ~11k to roughly the in-flight send count.

3.2 Raw-bytes lane through NAPI (P1 — the core fix)

Add byte-oriented native entry points alongside the string ones:

send_raw(port, host, payload: JsBuffer, cb)
send_with_header_raw(port, host, version, headerStr: JsString, payload: JsBuffer, cb)
multi_send_with_header_raw(ports, hosts, version, headerStr, payload: JsBuffer, cb)

• Neon supports JsBuffer natively; Rust gets a byte slice without any encoding step.
• Wire format: extend the existing versioned header (header_factory.rs) with a frame layout of header_len + header_bytes + payload_len + payload_bytes. The envelope fields that matter (UUID, msgDir, sender info) ride in the small header; the payload is opaque bytes.
• Receive side: the listener hands JS (headerObj, payloadBuffer) — the payload crosses NAPI once as a Buffer and is consumed directly by shardus-core's typed readers. No JS string, no JSON.parse, no base64.
• shardus-core: network.tellBinary/askBinary pass wrappedReq.getBuffer() straight through instead of {route, payload} (route moves into the header or the first bytes of the payload, which the binary protocol already encodes via TypeIdentifierEnum).

CPU eliminated per message, per direction (everything in §2.2 steps 3–7 except one copy):

• base64 encode/decode — ~1 byte-op per payload byte, twice per hop, gone
• envelope JSON.stringify/JSON.parse including the megastring — gone
• V8 string ↔ Rust String UTF-8 conversions of the full message — replaced by one Buffer handoff
• intermediate string/Buffer allocations → direct GC reduction
• ~33% fewer bytes through every remaining copy, the socket, and the network

Compression bonus: gzip/brotli (already implemented in Rust) compresses raw binary meaningfully better and cheaper than base64 text — base64 destroys byte-level patterns. Raw lane + selective compression for large payloads (account data tells) is strictly better than today on both CPU and bandwidth.

3.3 Envelope diet (P2)

Move augData bookkeeping (UUID, PORT/ADDRESS, msgDir) into the binary header; make the four timestamp fields (sendTime, receivedTime, replyTime, replyReceivedTime) optional/trace-only. Today they're stringified into every message. Small, but it's per-message and makes the header fixed-size and parseable in Rust without JSON.

3.4 Serialize once per batch (P3)

For multi-recipient sends, encode the frame once and hand Rust the recipient list — the combined path (useCombinedTellBinary) already has this shape; make it the default in shardus-core and extend the raw lane's multi-send accordingly. Eliminates the per-recipient re-stringify (currently ×3 for gossip batches, ×9 for receipt broadcasts on the per-node path).

3.5 Receive-side dedup below the JS boundary (P4 — optional, larger)

Gossip delivers each tx and each receipt ~3× per node (measured); every duplicate currently crosses NAPI, becomes a string, and gets parsed before JS-level dedup discards it. With the raw lane in place, a Rust-side LRU keyed on payload hash could drop duplicates before they cross into JS at all, cutting inbound NAPI crossings and JS handler invocations by roughly half. Needs care (dedup semantics belong to the gossip protocol), so this is a follow-on, not part of the core migration.

---

4. Rollout and verification

Compatibility: the raw frame is a new header version. Nodes advertise/accept both formats during a transition window (the header-version mechanism exists for exactly this); shardus-core gates sending on a config flag (e.g. p2p.useRawBinaryNet) flippable per-network. Old-format receive support is retained until the network's minimum version passes the cutoff.

Ship order: 3.1 first (independent, immediately verifiable, changes no wire format) → 3.2 + 3.3 together behind the flag → 3.4 in the same header rev if timing allows → 3.5 later.

Verification at each step, using the capture set we already have:

Metric	Today (local 10 TPS / cloud 12 TPS)	Expected after 3.1	Expected after 3.2–3.4
Pending Timeout objects	~11,300	≈ in-flight sends (<100)	same
arrayBuffers held	94MB / ~490MB	<30MB / <100MB	lower still (fewer allocs)
Suspended _wrappedSendAug frames	~4,900	~0	~0
JS busy in JSON/parse frames	~16% of busy (10-node floor; expected higher at 32 nodes)	similar	near zero for internal messages
Bytes on wire per message	payload×1.33 + JSON envelope	same	payload×1.0 + ~fixed header
Process CPU at fixed TPS	29% / 56% (contended hosts — baselines, not calibrations)	somewhat lower (GC)	target ≥25–30% reduction vs own like-for-like baseline

Plus one microbenchmark in lib-net CI: round-trip a 1KB / 10KB / 100KB payload through old vs raw path, asserting the raw path does no base64/JSON work (can be checked by allocation counts).

Honest caveats on the numbers: every percentage in this document is environment-specific. The local profile came from a laptop running all 10 node processes alongside normal desktop software; the cloud captures came from hosts running multiple nodes each. Neither is production-representative hardware, so treat all absolute CPU figures as directional — what is robust is the identity of the frames (base64/JSON/copy work exists on every message), the heap evidence (pinned payloads, pending timers), and the JS-idle-vs-process-busy ratio. The 16%-of-JS-busy figure is additionally a floor because 10-node conditions minimize message count and payload size; the same code path at 32 nodes does strictly more of this work per tx. Prerequisites before/after each change ships: (a) capture a .cpuprofile + heap snapshot from a 32-node network node on representative hardware, ideally one node per host; (b) compare only like-for-like runs on identical setups; (c) re-baseline after 3.1 lands, since part of today's measured cost is GC amplification from the timer leak.