Skip to content

feat(serder): #79 pluggable serialization backend — seam + DirectJson zero-copy writer#141

Merged
joeldsouzax merged 5 commits into
mainfrom
feat/79-serialization-backend-seam
Jul 10, 2026
Merged

feat(serder): #79 pluggable serialization backend — seam + DirectJson zero-copy writer#141
joeldsouzax merged 5 commits into
mainfrom
feat/79-serialization-backend-seam

Conversation

@joeldsouzax

Copy link
Copy Markdown
Contributor

Summary

Implements the #79 design (reviewed in #138): serialization becomes a pluggable component behind the EventSerializer seam, with serde_json as the unchanged default and a hand-rolled DirectJson backend selectable per call via serialize_with. Zero new dependencies.

The seam (commit 1)

SAID/version orchestration is written once in serialize_with: render once with a placeholder SAID + zero-size version string, backpatch the measured size in place, hash, splice the SAID into the EventLayout slots (both fixed-width). Backends only control how bytes are produced. The old per-ilk pipelines refactor behind the seam with unchanged public signatures; as a side effect even the serde_json reference drops from 3 full JSON renders per event to 1 render + 2 in-place patches.

DirectJson backend (commit 2)

Writes the five fixed event grammars straight into the caller's buffer — field names/framing as compile-time constants, values through an RFC 8259 escaper, slot offsets recorded while writing. No Value tree, no intermediate String.

Also fixes a panic found while mirroring the threshold renderer: tholder_to_json hit 0/0 integer division on a (0, 0) weight, and parse_weight accepted "0/0" from the wire — so a crafted event with a valid SAID deserialized fine and panicked on re-serialization. parse_weight now rejects zero denominators; weight rendering is division-free and shared by both backends. Bug probes red-before/green-after.

Conformance gates (commits 2–3)

  • Cross-backend proptests per ilk: DirectJson output must be byte-identical to SerdeJson over generated events (boundary sn/bt/thresholds, all five seal variants, both identifier derivations)
  • Escaper oracle: property-tested against serde_json's own string encoder
  • Cross-path: direct output SAID-verifies through the unchanged read path
  • Allocation budget (tests/serder_allocation.rs): counting allocator asserts direct < serde_json allocations
  • Existing round-trip / kel-chain / keripy differential suites pass untouched (1872 tests)

Benchmarks (benches/serder.rs, CodSpeed-wired)

Local aarch64-darwin, both figures include the fixed Blake3 hashing cost:

bench serde_json direct
icp 4.08 µs 2.32 µs 1.76× faster
ixn + 16 anchors 7.33 µs 4.20 µs 1.75× faster

Breaking change (0.x minor)

New SerderError::InvalidEventLayout variant (backend-reported layout inconsistencies surface as typed errors, never a corrupt frame). Everything else is additive: EventSerializer, EventRef, EventLayout, SerdeJson, DirectJson, serialize_with.

Follow-ups (per design §3.3/§3.5)

Closes #79

🤖 Generated with Claude Code

joeldsouzax and others added 3 commits July 10, 2026 17:12
…n behind EventSerializer

Implements the #79 design (docs/superpowers/specs/2026-07-10-79-
serialization-backend-seam-design.md §3.2): the SAID/version orchestration
moves into serialize_with — render once with a placeholder SAID and
zero-size version string, backpatch the measured size in place, hash,
splice the SAID into the reported EventLayout slots. Backends implement
EventSerializer and only control how bytes are produced.

The serde_json path becomes the SerdeJson reference backend: each per-ilk
module keeps its build_*_json field logic as render_json, and the public
serialize_* signatures are unchanged. Byte identity is proven by the
untouched round-trip, kel-chain, and keripy differential suites (1862
tests green). As a side effect the reference backend itself drops from
three full JSON renders per event to one render + two fixed-width
in-place patches.

New public surface (additive): EventSerializer, EventRef, EventLayout,
SerdeJson, serialize_with. New error variant InvalidEventLayout (breaking
per error-variant policy; called out in CHANGELOG).

Refs #79

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second half of the #79 seam: DirectJson renders the five fixed event
grammars straight into the caller's buffer (field names and framing as
compile-time constants, values through an RFC 8259 escaper), recording
slot offsets as it writes — no serde_json::Value tree, no intermediate
String per render. Zero new dependencies.

Also fixes a panic found while mirroring tholder_to_json: a (0, 0)
weight hit 0/0 integer division, and parse_weight accepted "0/0" from
the wire — so a crafted event with a valid SAID deserialized fine and
panicked on re-serialization. parse_weight now rejects zero
denominators (typed error) and weight rendering is division-free
(weight_to_string, shared by both backends).

Conformance gates in-module: per-ilk cross-backend proptests assert
byte-identical output between DirectJson and the SerdeJson reference
(boundary sn/bt/threshold values, all five seal variants, both
identifier derivations); the escaper is property-tested against
serde_json's own string encoder; a cross-path test SAID-verifies
direct output through the unchanged read path.

Refs #79

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- tests/serder_allocation.rs: counting-allocator safeguard asserting the
  direct backend allocates strictly less than the serde_json reference
  for the same event (the win is behaviorally invisible, so conformance
  tests cannot catch an allocation regression).
- benches/serder.rs: serialize benches through both backends on
  identical deterministic fixtures (representative icp, 16-anchor ixn).
  Local numbers (aarch64-darwin): icp 4.08us -> 2.32us, ixn16
  7.33us -> 4.20us (~1.75x, including the fixed Blake3 cost in both).

Refs #79

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codspeed-hq

codspeed-hq Bot commented Jul 10, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 21 untouched benchmarks
🆕 4 new benchmarks
⏩ 21 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
🆕 icp_direct N/A 24.7 µs N/A
🆕 icp_serde_json N/A 47.6 µs N/A
🆕 ixn16_direct N/A 36.5 µs N/A
🆕 ixn16_serde_json N/A 75.1 µs N/A

Comparing feat/79-serialization-backend-seam (2747123) with main (c6f1448)

Open in CodSpeed

Footnotes

  1. 21 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

…nds, framing, boundaries

Closes the untested-invariant gaps in the #79 seam, each as a
failing-when-broken test:

- Hostile-backend boundary: EventSerializer is public, so serialize_with
  must survive any reported layout — out-of-bounds size/SAID/prefix
  slots, wrong-width slots, reversed ranges all surface as typed
  InvalidEventLayout, never a panic or corrupt frame (5 tests).
- extend_with_layout framing: rejects renders without the zero-size
  version head, with a nonzero size, missing the SAID placeholder, or
  missing the second placeholder on double-SAID events; base-offset
  arithmetic proven by rendering into a prefilled buffer (5 tests).
- Append semantics: both backends report absolute slot offsets when the
  caller's buffer is non-empty (2 tests).
- weight_to_string exact mapping table incl. division-free 0/0, 1/0,
  unreduced 2/4, and u64::MAX collapse.
- EventRef: ilk / is_double_said / From<&KeriEvent> mappings for all
  five ilks (previously untested public API).
- Escaper: deterministic probe covering every escape class (quote,
  backslash, five short escapes, \u00xx fallbacks, DEL boundary,
  multi-byte UTF-8) plus an any::<String>() oracle proptest — the ".*"
  regex strategy under-samples control characters.
- Tholder boundaries: empty clause list / empty clause byte-identical
  across backends; proptest strategy widened to generate empties.

Refs #79

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@joeldsouzax joeldsouzax enabled auto-merge (squash) July 10, 2026 16:35
@joeldsouzax joeldsouzax merged commit 6e9ba71 into main Jul 10, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pluggable zero-copy serialization backend (swap serde for an optimized serializer)

1 participant